+

username being your github username, branch being the branch you have been working on.

+ +

You should now be able to see listed below, all the changes that you wish to contribute. If everything looks as it should, click ‘Create pull request’.

+ +

Give your pull request a title and description, then click ‘Create pul request’. You have now contributed your pages to the OpenSpending Community site :)

diff --git a/examples/openspending/content/about/meta/contribute/editing.html b/examples/openspending/content/about/meta/contribute/editing.html new file mode 100644 index 00000000..5e8f4832 --- /dev/null +++ b/examples/openspending/content/about/meta/contribute/editing.html @@ -0,0 +1,103 @@ +--- +lang: en +title: Editing a page +authors: + - Sam Smith +section: meta +--- + +

If you haven’t done so already, the first thing you need to do is head over to Github and create your free account.

+ +

There are three steps to editing a page. First you need to locate the page you wish to edit. There are a couple of ways to do this. Method A is probably the simplest, and most likely way you’ll do it. Method B will serve as a primer for the next section: Adding a page.

+ +

1: Locate the page

+ +

Method A: Browse the website

+ +

While reading any section of the Handbook you’ll see an 'Edit this page' link in the bottom left of the page. Following this link will take you directly to an editable version of that page. Easy huh?

+ +
+
Note
+

When the editable page opens it will (most likely) contain a message saying “You're editing a file in a project you don't have write access to”. If this is your first edit to The Open Data Handbook it will say “We've created a fork of this project for you to commit your proposed changes to”. This is normal and part of the workflow.

+
+ +

Method B: Browse the Github repository

+ +

The entire file structure of this site can be browsed on Github. For example, the root of the site is here, and the English language handbook section is here. It’s helpful to understand that the page URLs correspond to the file structure you see here. So, if you wanted to edit the Handbook introduction page, given that it’s URL is {{ site.url }}/en/introduction/ we know this file can be found in the en directory with the filename introduction.md Note: the extension (.md) is stripped from the URL. Following these links you should see a preview of the page you wish to edit. From here click the edit icon [pencil icon] to start editing.

+ +
+
Pro Tip!
+

Press t on any tree or blob page to launch the file finder.

+
+ +

2: Make your changes

+ +With the editable content in front of you, you’re probably either thinking “great, let’s get editing”, or “hang on, this looks a bit weird”. In case it’s the latter, let’s have a closer look. + +The first thing to recognise is the ‘Front Matter’, which will look like this: + +
+---
+title: Introduction
+---
+
+ +

The front matter must be the first thing in the file, must adhere to the above syntax, and must be set between triple-dashed lines. Numerous variables can be set here, but you’ll usually just need title. The title set here will be used as the main heading for the page, as well as in the browser tab.

+ +

The other important thing to recognise is the Markdown syntax. For example, where you see a line commencing with two hash marks:

+ +
+##Do you know exactly how much of your tax money is spent on street lights?
+
+ +

This is the Markdown way of creating a level two heading. On the site it will be outputted like so:

+ +

Do you know exactly how much of your tax money is spent on street lights?

+ +

Another common formatting requirement is bullet points, or lists. These are achieved in Markdown by using asterisks, like so:

+ +
+* civil servants
+* journalists
+* politicians
+
+ +

giving you:

+ +
    +
  • civil servants
    • +
  • journalists
    • +
  • politicians
    • +
+
+

Links are created like so:

+ +
+Give your data a home at the [Datahub](http://datahub.io/).
+
+ +

result:

+ +

Give your data a home at the Datahub.

+ +
+
Pro Tip!
+

To get a link to a specific heading on this site, hover over it then click the section icon [section icon]. This will put the URL into your address bar.

+
+ +

More Markdown examples can be found here, and a more detailed overview here.

+ +

If you are unsure of your markup while editing, you can switch to the preview tab [eye icon] Preview changes to see how it will be rendered.

+ +
+
Note
+

The Github previews will look stylistically different from the live site. A different font will be used for example.

+
+ +

Once you are happy with your changes, add a summary of what you've changed in the field below the editable text. Then click ‘Propose file change’.

+ +

3: Make a pull request

+ +

You will now be presented with a ‘pull request’ form. So far, the changes you have made are to your own copy, or fork of the handbook. A pull request simply sends a request to the authors/maintainers of the live handbook, asking them to include your changes - and put them live! Add any comments you have for the handbook team, then press ‘Create pull request’.

+ +

Your work here is done :) If you need to make related changes though, any new commits pushed to your branch will automatically be added to the pull request.

diff --git a/examples/openspending/content/about/meta/contribute/index.md b/examples/openspending/content/about/meta/contribute/index.md new file mode 100644 index 00000000..985a3736 --- /dev/null +++ b/examples/openspending/content/about/meta/contribute/index.md @@ -0,0 +1,76 @@ +--- +title: Contributing to this site +authors: + - Sam Smith +lang: en +section: meta +--- + +Thank you for your interest in in helping to build the OpenSpending +community site. We warmly welcome comments, corrections and additions, +as well as suggestions for additional sections and areas to +examine. For general discussion about +[OpenSpending](https://openspending.org/), please visit +[our forums](https://discuss.okfn.org/c/openspending). To jump in with +improvements and additions, read on. + +## How this site works + +In order to contribute, you need a little insight of how things work +under the hood. We’re not going to go into too much detail here, but +these are the three components you need some understanding of: + +- GitHub +- Jekyll +- Markdown + +### GitHub + +#### What is it? + +GitHub is a web-based repository hosting service, which amongst other +things offers revision control and source code management via a +web-based graphical interface. + +#### Why should I care? + +Any changes you wish to make, whether they be edits to an existing +page, or creating a new one, will most likely be done via the GitHub +website (it is also possible to download and edit the files on your +local machine, instructions for this method will be added in the +future). All the files for this site can be browsed and edited the +GitHub website. You will need to [sign up](https://github.com/) for a +(free) GitHub account. For full instructions, see +[Editing a page](./editing/). + +### Jekyll + +#### What is it? + +Jekyll is a static site generator, which allows us to host websites +based on our GitHub repositories. Jekyll takes the content, renders +Markdown, and produces a complete, static website ready to be viewed +on the web. + +#### Why should I care? + +All you really need to know about Jekyll is the method it uses to +include metadata (ie. page title). Each page needs to start with a +section it calls Front Matter, containing the page title. An example +is provided in the [Adding a page](./adding/) section. + +### Markdown + +#### What is it? + +Markdown is a markup language with plain text formatting, designed so +that it can be converted to HTML. Markdown can be used to create rich +text using a plain text editor. + +#### Why should I care? + +Markdown is your key to formatting the text your provide for this +site. By learning a few intuitive rules you’ll be able to ensure your +text is formatted with headings, list, quotes etc, without writing any +HTML. For examples, head to the +[Markdown]({{site.baseurl}}/meta/contribute/markdown-examples/) section. diff --git a/examples/openspending/content/about/meta/contribute/markdown-examples.md b/examples/openspending/content/about/meta/contribute/markdown-examples.md new file mode 100644 index 00000000..0b58360d --- /dev/null +++ b/examples/openspending/content/about/meta/contribute/markdown-examples.md @@ -0,0 +1,213 @@ +--- +lang: en +title: Markdown Examples +section: meta +--- + +* TOC +{:toc} + + * TOC + {:toc} + +This is a paragraph. + + This is a paragraph. + + + +Header 1 +======== + +Header 2 +-------- + + Header 1 + ======== + + Header 2 + -------- + + + +# Header 1 + +## Header 2 + +### Header 3 + +#### Header 4 + +##### Header 5 + +###### Header 6 + + # Header 1 + + ## Header 2 + + ### Header 3 + + #### Header 4 + + ##### Header 5 + + ###### Header 6 + + + + + + +> Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus. Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus. + + > Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus. Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus. + + + +> ## This is a header. +> 1. This is the first list item. +> 2. This is the second list item. +> +> Here's some example code: +> +> Markdown.generate(); + + > ## This is a header. + > 1. This is the first list item. + > 2. This is the second list item. + > + > Here's some example code: + > + > Markdown.generate(); + + + + + + +* Red +* Green +* Blue + +~~~ +* Red +* Green +* Blue +~~~ + + + +1. Buy flour and salt +1. Mix together with water +1. Bake + +~~~ +1. Buy flour and salt +1. Mix together with water +1. Bake +~~~ + + + +Paragraph: + + Code + + + + Paragraph: + + Code + + + +* * * + +*** + +***** + +- - - + +--------------------------------------- + + * * * + + *** + + ***** + + - - - + + --------------------------------------- + + + +This is [an example](http://datahub.io/) link. + +[This link](/about/) is internal. + +This is [an example] [ok] reference-style link. + +[ok]: https://okfn.org/ + + This is [an example](http://datahub.io/) link. + + [This link](/about/) is internal. + + This is [an example] [ok] reference-style link. + + [ok]: https://okfn.org/ + + + +*single asterisks* + +_single underscores_ + +**double asterisks** + +__double underscores__ + + *single asterisks* + + _single underscores_ + + **double asterisks** + + __double underscores__ + + + +This paragraph has some `code` in it. + + This paragraph has some `code` in it. + + + +![Alt Text](http://placehold.it/200x50 "Image Title") + + ![Alt Text](http://placehold.it/200x50 "Image Title") + + +| Tables | Are | Cool | +| ------------- |:-------------:| -----:| +| col 3 is | right-aligned | $1600 | +| col 2 is | centered | $12 | +| zebra stripes | are neat | $1 | + + | Tables | Are | Cool | + | ------------- |:-------------:| -----:| + | col 3 is | right-aligned | $1600 | + | col 2 is | centered | $12 | + | zebra stripes | are neat | $1 | + +I bet you'd like more information about this sentence [^1]. + +[^1]: Well lucky for you, I've included more information in a footnote. + +~~~ + I bet you'd like more information about this sentence [^1]. + [^1]: Well lucky for you, I've included more information in a footnote. +~~~ diff --git a/examples/openspending/content/about/meta/contribute/translate-glossary.html b/examples/openspending/content/about/meta/contribute/translate-glossary.html new file mode 100644 index 00000000..cd07f6cd --- /dev/null +++ b/examples/openspending/content/about/meta/contribute/translate-glossary.html @@ -0,0 +1,65 @@ +--- +lang: en +title: Translating the glossary +authors: + - Mor Rubinstein +section: meta +--- + +

What's new in the glossary

+

In the old version of the handbook, the glossary was one page with all of the term in it. In the new version, we gave each glossary a webpage for better referencing and linking.<\p> +Glossaries that were translated in the old version of the handbook have been transfered to the new site. Please checkCheck if your language has an old version of the glossary - +. You can find them here + +

If you do have an old version translated, follow these steps:

+

The old glossary format does not allow linking from the new version of the guide, and you will need to transfer the term to the new format.

+ +

1. Create a new folder for the term

+ +

Under your language folder, Follow the breadcrumb trail {{ site.github_repo }} / glossary / es /, to the right of which is a plus symbol +. create a folder for each term by pressing on the '+' sign and type the term name in English. The folder names should be in lower-case letters with dashes ‘-’ instead of white spaces. Add a ‘/’ in the end of the name to create a new folder.

+ +

2. Translate the term

+

open a new index.md file by clicking on the '+'.

+ +In the text editor below add the front matter: +
+---
+section: terms
+lang: en
+title: Bulk
+---
+
+ +

Change the 'lang' field to your language code. Change the title to the term title in YOUR language.

+

Below the front matter copy the term from the old glossary.

+ +

3. Make a pull request.

+ +

All done! Keep doing this until all terms got their own folder and page.

+ +

If you have never translated the glossary before, follow the these steps:

+ +

Copy the English Glossary

+ +

Copy the English terms directory into your target language directory in the glossary folder. This step can not be done through the Github website and you will have to fork the handbook for you machine. Information about forking and cloning can be find here . Notice, some languages have already been moved and translated. Check the folder to make sure you are not overwriting someone’s work.

+ +

Edit the term

+ +

Choose a term and open the index.md file.

+

The front matter looks as follows:

+ +
+---
+section: terms
+lang: en
+title: Bulk
+---
+
+ +

Change the lang field to your language code.

+

Change the title to the term title in YOUR language.

+

In the text editor, below the front matter, enter your translation to the term.

+ +

3. Submit changes through a pull request.

+ +

All done! Thank you for your help!

diff --git a/examples/openspending/content/about/meta/contribute/translate-guide.html b/examples/openspending/content/about/meta/contribute/translate-guide.html new file mode 100644 index 00000000..d2f81da0 --- /dev/null +++ b/examples/openspending/content/about/meta/contribute/translate-guide.html @@ -0,0 +1,41 @@ +--- +lang: en +title: Translating the guide +authors: + - Mor Rubinstein +section: meta +--- + +Translating the guide is easy, no need to any other software, all you need is a github account! +Some languages already have translated version of the guide. If you don't have a version in your language, here is how to do it. + +

1. Create a new language folder

+ +In github, under the breadcrumb - {{ site.github_repo }} / guide/ there will be a '+' sign. Click on it and enter your your two letter languages code. Add a dash ('/') after the two letter to create a folder. + +

2. Create a page folder

+ +Now you will see you languages code and a '+' sign on the breadcrumb. Add a the page name that you want to translate in __English__ and add a dash at the end. + +

3. translate the content

+ +You will now see a new '+' sign. Add the file name 'index.md'. +In the text editor add the following front matter: +
+---
+section: guide
+lang: Your language two letter code
+title: The title in your language
+---
+
+ +Translate as usual. + +

4. Create a pull request

+If all good, we will add it to the site. + +Repeat for other parts of the guide if needed. + +That's it, you are all done! + +Thank you for helping us to make the guide accessiable to others! diff --git a/examples/openspending/content/about/meta/index.md b/examples/openspending/content/about/meta/index.md new file mode 100644 index 00000000..d6f3eb41 --- /dev/null +++ b/examples/openspending/content/about/meta/index.md @@ -0,0 +1,35 @@ +--- +title: About This Site +--- + +If you are a member of the OpenSpending community---whether you +are a data publisher, work on building or testing code, or develop +stories and visualizations using OpenSpending data---this site belongs +to you. + +This site is a place to showcase the OpenSpending community's work and +to share its resources. It includes: + +* information about community [events][3b] +* a collection of [resources][2] on spending data +* guidance on [how to get involved][3] with the community + +[This site](https://github.com/openspending/community.openspending.org/) +is hosted on [GitHub Pages](https://pages.github.com/) and built using +[Jekyll](http://jekyllrb.com/), a static site generator. Interested +readers are encouraged to contribute changes, fixes, corrections by +[raising an issue](https://github.com/openspending/community.openspending.org/issues/new?title=Bug&body=I%27m%20having%20an%20issue%20with...) +on the site's +[issue tracker](https://github.com/openspending/community.openspending.org/issues). +For more detail on how to contribute, see the Contribution Guide +below. + +- [Contribution Guide](./contribute/): Detailed information on how to contribute changes to the site +- [Recent Changes](./changes/): A list of recent changes to the site +- [Media](./media/): Downloadable OpenSpending images and other media + +[1]: {{ site.baseurl }}/blog/ +[2]: {{ site.baseurl }}/resources/ +[3]: {{ site.baseurl }}/get-involved/ +[3b]: {{ site.baseurl }}/events/ +[4]: {{ site.baseurl }}/help/ diff --git a/examples/openspending/content/about/meta/media/index.md b/examples/openspending/content/about/meta/media/index.md new file mode 100644 index 00000000..7c4f6eee --- /dev/null +++ b/examples/openspending/content/about/meta/media/index.md @@ -0,0 +1,22 @@ +--- +title: Media +section: meta +--- + +## OpenSpending Square Logo + +![OpenSpending Square](/img/OpenSpending_400x400.jpg) + +## OpenSpending Text Logo + +### Normal + +![Normal](http://assets.okfn.org/p/openspending/img/openspending-logo.png) + +### Small + +![Small](http://assets.okfn.org/p/openspending/img/openspending-logo-s.png) + +### Large + +![Large](http://assets.okfn.org/p/openspending/img/openspending-logo-l.png) diff --git a/examples/openspending/content/about/openspending-first-steps.md b/examples/openspending/content/about/openspending-first-steps.md new file mode 100644 index 00000000..0da6739b --- /dev/null +++ b/examples/openspending/content/about/openspending-first-steps.md @@ -0,0 +1,102 @@ +--- +section: about +lead: true +title: 'OpenSpending: first steps' +authors: +- Neil Ashton +redirect_from: +- "/openspending-first-steps" +--- +
+
+

Welcome to OpenSpending

+
+
+
+
+
+
+
+00-visualize +
+
+

Upload and visualize data

+Upload any kind of financial data to OpenSpending and explore it with our built-in interactive visualizations. Users publish budgets, procurements, spending data and even public employee salaries. + +Use our widgets to embed your visualization on your own website. + +Upload a dataset + +
+
+
+
+
+
+
+
+01-explore +
+
+

Explore the database

+OpenSpending holds nearly 16 mio. transactions from more than 300 datasets across more than 70 countries. + +Browse the list of datasets and learn about public finances from around the world, or browse our map of cities on OpenSpending. + +Search transactions or look at the datasets + +
+
+
+
+
+
+
+
+02-extend +
+
+

Create your own dataviz with our API

+Create your own visualizations with the OpenSpending API using libraries like D3.js, jit and Raphael. + +You can even create a satellite site while still using OpenSpending as a backend. + +Look at the API + +
+
+
+
+
+
+
+
+03-community +
+
+

Join the community!

+OpenSpending is open source software built and run by a community of volunteers. + +Join our mailing list and share what you are creating with OpenSpending! + +Join the spending community or Become a developer + +
+
+
+
+
+ diff --git a/examples/openspending/content/about/presentation-introduction.md b/examples/openspending/content/about/presentation-introduction.md new file mode 100644 index 00000000..14adef86 --- /dev/null +++ b/examples/openspending/content/about/presentation-introduction.md @@ -0,0 +1,13 @@ +--- +section: about +lead: true +title: "Introduction to OpenSpending: Mapping the Money" +presentation: true +authors: +- Anders Pedersen +redirect_from: /about/community-site/slide-deck-introduction/ +--- + +
+ +
diff --git a/examples/openspending/content/about/presentations.md b/examples/openspending/content/about/presentations.md new file mode 100644 index 00000000..1e8f399f --- /dev/null +++ b/examples/openspending/content/about/presentations.md @@ -0,0 +1,17 @@ +--- +title: Presentations +section: about +--- + +{% for presentation in site.pages %}{% if presentation.presentation %} +[{{ presentation.title }}]({{ presentation.url }}) +{% if presentation.authors %} +
Written by +
    +{% for author in presentation.authors %} +
  • {{ author }}
    • +{% endfor %} +
+
+{% endif %} +{% endif %}{% endfor %} diff --git a/examples/openspending/content/about/steering-group/index.md b/examples/openspending/content/about/steering-group/index.md new file mode 100644 index 00000000..9b73a259 --- /dev/null +++ b/examples/openspending/content/about/steering-group/index.md @@ -0,0 +1,126 @@ +--- +section: about +lead: true +title: Steering Group +redirect_from: + - "/about/governance/members-of-the-steering-group/index.html" + - "/about/governance/members-of-the-steering-group/" +--- + +The steering group oversees the project and represents its major stakeholders. This group takes responsibility for maintaining the integrity of the project, setting project policies, representing the project in relation to third parties etc. + +The steering group's specific responsibilities include: + +* **Finances**: overseeing and managing (where relevant) any project finances and resources. +* **Branding**. Overseeing Twitter account, website, etc. +* **Recognition** of satellite sites as part of the OpenSpending community and use of the OpenSpending name. +* **Partnerships** and collaborations with other communities and projects. +* **Setting policy** for data licensing and other matters, as necessary. +* **Advocacy guidelines**, as appropriate. +* **Legal matters**: for example, removal of datasets + +## Meetings + +The Steering Group meets approximately once a quarter via an online conference call. + +The [living minutes of the OpenSpending Steering Group are online](https://docs.google.com/a/okfn.org/document/d/1jCB-RquGYeW9mm466ViucMRjggbCxa0pGZDH5JThcRc/edit). + +## Appointment of the Steering Group + +Anyone who has contributed to the OpenSpending project (whether in code, data, or content) may put themselves forward for membership in the steering group. The existing steering group members will consider the application and decide whether or not to accept. In the future, once the concept of OpenSpending membership is better established, this decision may be made by a vote among OpenSpending members. + +Membership in the steering group is for a term of two years. Multiple consecutive terms are permitted. If members resign during their two-year term, nominations for a replacement will be solicited from the community. + +## Members + +### Anna Alberts, Open Knowledge Germany, Chair + +Based in Berlin, Anna Alberts works as a Project Manager for the EU +research project OpenBudgets.eu at Open Knowledge Foundation Germany. + +Anna studied International Development and International Relations, +and worked as a policy officer for the Strategy Unit at the Ministry +of Foreign Affairs in the Netherlands, focusing on geopolitics and +(open) data. + +### Justin Arenstein, International Center for Journalists +{: .person-name} + +![Justin Arenstein](http://www.icfj.org/sites/default/files/imagecache/medium/JustinWebsite_0.JPG) +{: .person-photo} + +Justin Arenstein is a Knight International Journalism Fellow who is +helping the African Media Initiative (AMI) to establish a digital +innovation program that supports experimentation in newsrooms across +Africa. AMI, the continent's largest association of media owners and +executives, is working with more than 600 of the most influential +media companies in both northern and sub-Saharan Africa. + +### Júlia Keserű, Sunlight Foundation +{: .person-name} + +Júlia Keserű is the International Policy Manager at the Sunlight +Foundation and oversees its international work. Coming from the +Hungarian transparency community, Júlia has been an advocate for open +government and an expert on open data issues with a special focus on +political finance and corruption. She has spoken internationally on +technology and transparency and regularly writes about the challenges +and the potential of the global open government movement. Júlia holds +a Masters degree from International Studies and studied political +sciences, international law, sociology and philosophy at Corvinus +University Budapest, Free University Berlin and the College for Social +Theories in Budapest. + +### Elena Mondo, International Budget Partnership +{: .person-name} + +![Elena Mondo, IBP]({{ site.baseurl }}/img/blog/2014/04/Elena-pic-223x300.jpg) +{: .person-photo} + +Elena joined the International Budget Partnership (IBP) in 2007. She +manages the Open Budget Survey, the only independent, comparative, and +regular measure of budget transparency and accountability around the +world. Prior to joining the IBP, she worked as a consultant for the +Organization for Economic Cooperation and Development (OECD), +coordinating research on budget practices and procedures in the OECD +and Latin American countries. Mondo holds a BA in international +economics and management from Bocconi University, and an MPA in public +and economic policy from the London School of Economics. + +### Oluseun Onigbinde, BudgIT +{: .person-name} + +![Oluseun Onigbinde](http://under40preneur.com/wp-content/uploads/2013/02/Seun-Onigbinde.jpg) +{: .person-photo} + +Oluseun Onigbinde is the Co-Founder of BudgIT, a Nigerian public data +visualisation startup. He is an Ashoka Fellow and Open Knowledge +Ambassador for Nigeria. + +### Anders Pedersen, Natural Resource Governance Institute +{: .person-name} + +Anders Pedersen is Open Data Program Officer at the Natural Resource Governance +Institute. He holds an MA degree in Political Science from University of +Copenhagen, and has worked in development and financial data journalism. + +### Federico Ramírez, Fundar +{: .person-name} + +[bio coming here] + +### Adam Stiles, Open Budget Oakland +{: .person-name} + +Adam is co-creator of openbudgetoakland.org and a +member of the City of Oakland's budget advisory committee. He's also a +news editor, builder, and outdoor preschool founder. + +### Mark Brough, Publish What You Fund +{: .person-name} + +### Cecile Le Guen, Open Knowledge +{: .person-name} + +Cecile Le Guen is Project Manager at Open Knowledge, involved in different fiscal open data projects. diff --git a/examples/openspending/content/about/tools.md b/examples/openspending/content/about/tools.md new file mode 100644 index 00000000..4b3b1c8c --- /dev/null +++ b/examples/openspending/content/about/tools.md @@ -0,0 +1,22 @@ +--- +title: OpenSpending tools +redirect_from: + - "/about/governance/" +--- + +### [OpenSpending Viewer](https://openspending.org/s/) + +The OpenSpending Viewer is a Javascript app that provides views over data uploaded to OpenSpending. It offers 8 different visualisations and a pivot table for analyzing the data + +### [OpenSpending Packager](https://openspending.org/packager/) +Via OpenSpending Packager fiscal data can be uploaded from alternate sources (csv, Excel, Google Sheets and Fiscal Data Package). Data and metadata can be uploaded in 4 simple steps. + +### [OpenSpending Admin](https://openspending.org/admin/) +OpenSpending Admin offers the possibility to administer a user account and the associated data packages that have been loaded to the platform. You can access it from the main Packager or Viewer once you create an account. + +### OpenSpending DataMine +The OpenSpending DataMine is an experimental feature for investigative analysis of data with direct access to read any part of the database. This feature can be used (and it is encouraged!), but currently it is not subject to further customisation. + +## What kind of data can I upload to OpenSpending? + +OpenSpending is designed to work with any dataset in CSV format containing government budget, spending information or any other fiscal data. To upload this data, we use data pipelines and the Fiscal Data Package. We recommend learning a bit more about it, even if you're not part of the technical team uploading the data. diff --git a/examples/openspending/content/blog/2011-03-01-where-does-my-money-go-goes-international-welcome-to-openspending.md b/examples/openspending/content/blog/2011-03-01-where-does-my-money-go-goes-international-welcome-to-openspending.md new file mode 100644 index 00000000..580bed11 --- /dev/null +++ b/examples/openspending/content/blog/2011-03-01-where-does-my-money-go-goes-international-welcome-to-openspending.md @@ -0,0 +1,17 @@ +--- +authors: +- friedrich +redirect_from: /2011/03/where-does-my-money-go-goes-international-welcome-to-openspending/ +title: "'Where Does My Money Go?' Goes international. Welcome to OpenSpending." +--- + +**This post is by [Friedrich Lindenberg](http://okfn.org/members/pudo), one of the developers working on OpenSpending.** + +Our primary goal has to be to grow WDMMG as an open platform, similar to Open Street Map: while on OSM you sketch out your local streets, WDMMG should become the place to upload and analyze your local or state governments spending. Therefore, our priority has to be providing the right tools to allow people to contribute to this effort themselves: either by loading data, annotating spending or visualizing it in custom ways. + + + +As such transparency is needed not only in the UK but all over the world, we want to re-label the data part of the site (what is now data.wdmmg.org) to the more international OpenSpending. This would both serve as an accessible means to handling financial data and as a backend to more specific sites, such as the UK's WhereDoesMyMoneyGo visualizations and Germany's OffenerHaushalt. + +I'd like to invite all of you to follow up on the remainder of our discussion, which is archived at + and to contribute your own thoughts. diff --git a/examples/openspending/content/blog/2011-03-20-uploading-data-to-openspending.md b/examples/openspending/content/blog/2011-03-20-uploading-data-to-openspending.md new file mode 100644 index 00000000..3668ccd0 --- /dev/null +++ b/examples/openspending/content/blog/2011-03-20-uploading-data-to-openspending.md @@ -0,0 +1,21 @@ +--- +authors: +- lucy +redirect_from: /2011/03/uploading-data-to-openspending/ +title: Uploading Data to OpenSpending +--- +The amount of datasets that are available on [OpenSpending.org](http://www.openspending.org) are growing fast and we want more! Currently the process looks like this: + +1. You give us data. +2. We look at it, try to understand it, possibly ask you some more questions. +3. We write a custom loader script to load the data. + +To make this process easier for us and faster for everybody, we offer an alternative process that requires a bit more work from you. But if you know how to transform your data to our CSV format, you will have your spending data online on OpenSpending more quickly and we can spend more time developing features! Here is how it works: + +1. You create a CSV file that is formatted according to our [CSV schema](http://wiki.openspending.org/CSV_Schema). [Here is a really simple example of a CSV file][csv_example]. +2. You use [our new web based uploader](http://www.openspending.org/sources/add) that automatically checks your CSV file for errors and stores it along with some meta data. +3. Contact us and we will do the final step and load the data into OpenSpending.org. + +[csv_example]: https://spreadsheets1.google.com/ccc?hl=en&key=t8rduOMdinCo0smZjQvQUow&hl=en#gid=0 + +The schema and this alternative process are by no means set in stone: any feedback is appreciated! Most important: if you have spending data, but can't provide it in our CSV format, don't worry and just contact us. We always prefer some data over no data! diff --git a/examples/openspending/content/blog/2011-10-27-thoughts-from-the-global-investigative-journalism-conference.md b/examples/openspending/content/blog/2011-10-27-thoughts-from-the-global-investigative-journalism-conference.md new file mode 100644 index 00000000..12f4b32d --- /dev/null +++ b/examples/openspending/content/blog/2011-10-27-thoughts-from-the-global-investigative-journalism-conference.md @@ -0,0 +1,69 @@ +--- +authors: +- lucy +redirect_from: /2011/10/thoughts-from-the-global-investigative-journalism-conference/ +title: Thoughts from the Global Investigative Journalism Conference +tags: +- Data Journalism +- Spending Stories +--- +**This post is by [Lucy Chambers](http://okfn.org/members/lucychambers), community coordinator at the Open Knowledge Foundation, and [Friedrich Lindenberg](http://okfn.org/members/pudo), Developer on OpenSpending. They recently attended the Global Investigative Journalism Conference 2011 in Kyiv, Ukraine, and in this post, bring home their thoughts on journalist-programmer collaboration...** + +# The conference + +The Global Investigative Journalism Conference must be one of the most intense yet rewarding experiences either of us have attended since joining the OKF. With topics ranging from human trafficking to offshore companies, the meeting highlighted the importance of long-term, investigative reporting in great clarity. + +With around 500 participants from all over the globe with plenty of experience in evidence gathering, we used this opportunity to ask many of them how platforms like OpenSpending can contribute, not only to the way in which data is presented, but also to how it is gathered and analyzed in the course of an investigation. + +# Spending Stories - the brainstorm + +As many of you will be aware, earlier this year we won a Knight News Challenge award to help journalists contextualise and build narratives around spending data. Research for the project, [Spending Stories](http://blog.okfn.org/2011/06/22/spending-stories-is-a-winner-of-the-knight-news-challenge/), was one of the main reasons for our trip to Ukraine... + +During the data clinic session as well as over drinks in the bar of "Hotel President" we asked the investigators what they would like to see in a spend analysis platform targeted at data journalists. Cutting to the chase, they immediately raised the key questions: + +### How will it support my work? + + +It was clear that the platform should support the existing journalistic workflow through publishing embargos, private datasets and note making. At the same time, the need for statistical and analytical heuristics to dissect the data, find outliers and visualize distributions was highlighted as a means to enable truly data-driven investigations of datasets. The goal in this is to distinguish anomalies from errors and patterns of corruption from policies. + +### What's in it for my readers? + + +With the data loaded and analyzed, the next question is what value can be added to published articles. Just like DocumentCloud enabled the easy embedding of source documents and excerpts, OpenSpending should allow journalists to visualize distributions of funds, embed search widgets and data links, as well as information about how the data was acquired and cleaned. + +### What do I need to learn to do it? + + +Many of those we spoke to were concerned about the complexity required to contribute data. The recurring question was: should I even try myself or hire help? It's clear that for the platform to be accessible to journalists, a large variety of data cleansing tutorials, examples and tools need to be at their disposal. + +We've listed the full brainstorm on the [OpenSpending wiki](http://wiki.openspending.org/Spending_Stories_Ideas#GIJC_Brainstorm) + +You can also see the mind map with concrete points below: + + + +## Hacks & Scrapers - How technical need data journalists be? + +In a second session, "Data Camp" we went through the question of how to generate structured data from unstructured sources such as web pages and PDF documents. We tried to emphasize the value of easily machine-readable data over less structured information by pointing to some examples on ScraperWiki. + +As we went through basic steps needed to scrape a web page, the questions began turning towards the purpose of the exercise: +> "So why do we need to learn how to scrape? Can't we just hire someone to do this for us?" + +Our answer went something like... + +> "Well, yes - you can actually, but..." + +...it may be a good idea to have some understanding of which data can be easily retrieved and what difficulties and errors you might encounter in the extraction process. This includes: + +1. Understanding the possibilities and limitations of various data structures on the web to understand how to approach programmers and what to ask for (and importantly, what it is reasonable to pay). +2. Understanding how to quality-check data extracted from the internet and where errors could be introduced. +3. Appreciating that programmers are expensive and that having a basic understanding of some of the principles behind screen scraping yourself could save your organisation quite a lot of money for simpler tasks + +The notes from the scraping session are available on this [pad](http://pudo.okfnpad.org/scrapetutu) + +### So how do I hire a hacker? + +The final thing that became blatantly apparent in sessions such as "Journalist or Programmer? Do Reporters Need to become Coders?" was that there is a huge void that needs to be bridged between the hacker and journalist world. If I had a pound for every time someone at the conference asked me how they could find a hacker, would be mighty happy. We pointed people in the direction of Hacks and Hackers meetings but there is clearly a need for a more extensive 'address' book of reliable contacts is obvious. + +I will attempt to pull together some of the thoughts we had about how to find (and trust!) your hacker in a separate post to address some of these needs. If you have further advice or anecdotes on this subject, please don't hesitate to get in contact via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/wdmmg-discuss). + diff --git a/examples/openspending/content/blog/2011-11-11-new-translation-documentation-for-openspending.md b/examples/openspending/content/blog/2011-11-11-new-translation-documentation-for-openspending.md new file mode 100644 index 00000000..163a1ec8 --- /dev/null +++ b/examples/openspending/content/blog/2011-11-11-new-translation-documentation-for-openspending.md @@ -0,0 +1,28 @@ +--- +authors: +- lucy +redirect_from: /2011/11/new-translation-documentation-for-openspending/ +title: New Translation Documentation for OpenSpending +--- +There's been a lot of demand for us to document the translation procedure for OpenSpending, so this is now up and live on the [wiki](http://wiki.openspending.org/Translations) + +For reference, I've also briefly included the steps here: + +## In order to translate OpenSpending, please follow the following steps: + +* Create an account on [Transifex](https://www.transifex.net/home/) +* Email info [at] openspending.org with your Transifex Username and ask to be added to the group for your language of translation. +* Proceed to the following [link](https://www.transifex.net/projects/p/openspending/resource/openspendinguipot/) +* Click 'Add a translation' and follow the instructions on-screen. + + +## When you have finished your translation... + +* Drop us an email to info [at] openspending.org and we will include it into our next release. + + +## Things to be aware of + +* With each new release of the code, you may need to update your translation, to make sure all the new commands are accounted for...***We are currently building up to a big code release and will inform the list when the strings are stable. If you are eager to get going, you may start translating, most of your translation should be preserved, but there will be a little additional work to do before the release.*** + +**Happy translating, drop me an email if you have any questions via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending).** diff --git a/examples/openspending/content/blog/2011-11-16-openspending-v0-11-released.md b/examples/openspending/content/blog/2011-11-16-openspending-v0-11-released.md new file mode 100644 index 00000000..0dec5c1f --- /dev/null +++ b/examples/openspending/content/blog/2011-11-16-openspending-v0-11-released.md @@ -0,0 +1,23 @@ +--- +authors: +- friedrich +redirect_from: /2011/11/openspending-v0-11-released/ +title: OpenSpending v0.11 Released +tags: +- Releases +--- +We are happy to announce the release of the latest version of OpenSpending. Most of our work has been to improve how we store and or organise spending data. Users will notice that the web frontend has been refreshed and is now much better integrated. + + + +## New features include + +* New backend using a conventional relational database allowing clean separation of datasets, and better scalability. The database backend is also much more familiar to developers than the previous backend + * [OpenSpending Developer Documentation](http://openspending.readthedocs.org/en/latest/index.html) +* Lots of documentation for API users and visualization hackers + * [OpenSpending Data Wrangler Documentation](http://openspending.org/help/api.html) +* New theme based on twitter's bootstrap framework +* Begun support for i18n/translation of the frontend +* Better validation of input data and model. + +Feedback on the new site and features are welcome. Please drop us a line via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2011-11-28-openspending-visualisations-featured-in-the-guardian.md b/examples/openspending/content/blog/2011-11-28-openspending-visualisations-featured-in-the-guardian.md new file mode 100644 index 00000000..cce32a5d --- /dev/null +++ b/examples/openspending/content/blog/2011-11-28-openspending-visualisations-featured-in-the-guardian.md @@ -0,0 +1,27 @@ +--- +authors: +- lucy +redirect_from: /2011/11/openspending-visualisations-featured-in-the-guardian/ +title: OpenSpending visualisations featured in the Guardian +--- +**This post is by [Lucy Chambers](http://okfn.org/members/lucychambers), Community Coordinator on OpenSpending.** + +On Friday, the Guardian Poverty Matters blog published [a piece on the Uganda visualisation](http://www.guardian.co.uk/global-development/poverty-matters/2011/nov/25/uganda-aid-confusion-analyse-spending?newsfeed=true) that the OpenSpending team had been working on with [Publish What You Fund](http://www.publishwhatyoufund.org/). + + + +## From the article + +> "The [Publish What You Fund campaign group](http://www.publishwhatyoufund.org/) and the [Open Knowledge Foundation](http://okfn.org) have now produced a [visualisation of Uganda's aid and budget data for 2003-2006](http://www.publishwhatyoufund.org/uganda/uganda-with-data.htm#/^/2004/~/aid-and-domestic-spending-in-uganda-br----ugx-), billed as the first time both sets of data have been displayed together in a way that is easy to explore. A quick look shows just how big a piece of the puzzle aid spending is – more than 50% of overall resources available in Uganda for 2005-2006. The vast majority of this $1.1bn in aid was spent directly by donors on various projects, with only a third given to the government to spend along with its domestic resources. Interestingly, aid money made up only a small proportion of resources for education, while accounting for the majority of resources for health, agriculture, water and the environment." + +## Busan Aid effectiveness meeting + +The release of the visualisation comes ahead of the [Busan aid effectiveness meeting](http://www.aideffectiveness.org/busanhlf4/) and highlights some of the key benefits of opening up spending data, both to the donor organisations and the governments of the recipient countries themselves: + +> "Four years ago, researchers at the [London-based Overseas Development Institute](http://www.odi.org.uk/) took up the enormous task of trying to figure out how dozens of donors were spending aid in Uganda, and how that compared with where the government was allocating its own resources. The results were striking: it turned out the Ugandan government was only aware of half the aid being spent in the country, despite routinely requesting this information from donors." + +It is hoped that visualisations such as these will make it easier to digest complex datasets of this type, where a government receives support from multiple sources. It is also hoped that discussions around the topic will result in the more timely and regular release of data to help highlight practices that will lead to aid money being most effectively spent. + +**Read the full Article in [the Guardian Poverty Matters blog](http://www.guardian.co.uk/global-development/poverty-matters/2011/nov/25/uganda-aid-confusion-analyse-spending?newsfeed=true).** + +**Have data similar to this you would like to create a similar visualisation for? Drop us an email via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending).** diff --git a/examples/openspending/content/blog/2011-12-05-how-spending-stories-spots-errors-in-public-spending.md b/examples/openspending/content/blog/2011-12-05-how-spending-stories-spots-errors-in-public-spending.md new file mode 100644 index 00000000..41fc6399 --- /dev/null +++ b/examples/openspending/content/blog/2011-12-05-how-spending-stories-spots-errors-in-public-spending.md @@ -0,0 +1,62 @@ +--- +authors: +- martin +redirect_from: /2011/12/how-spending-stories-spots-errors-in-public-spending/ +title: How Spending Stories Spots Errors in Public Spending +tags: +- Data Journalism +- Spending Stories +--- +*This article was originally published on [MediaShift Idea Lab](http://www.pbs.org/idealab/2011/12/how-spending-stories-spots-errors-in-public-spending328.html) and was co-written by Martin Keegan, project lead for Spending Stories and Lucy Chambers, Community Coordinator for OpenSpending.* + +How public funds should be spent is often controversial. Information about how that money has already been spent should not be ambiguous at all. People arguing about the future will care about the present, and if data about past or present public spending is available, many will certainly look at it. When they do, occasionally they will find errors, or believe themselves to have found errors. + +[OpenSpending](http://openspending.org/), which aims to track every (public) government and corporate financial transaction across the world, encourages users to: + + * augment the existing spending database with additional sources of data + * use that data -- e.g., to write evidence-based articles and formulate informed decisions about how their society is financed. + +[Spending Stories is our effort](http://www.pbs.org/idealab/2011/09/spending-stories-to-help-journalists-analyze-spending-data258.html) to make OpenSpending a natural way to do data journalism about public spending. + +openspending.jpg + +## The Problem + +**FACT 1:** Errors occur in data, no matter how official the source. + +**FACT 2:** Data wrangling (manipulating or restructuring datasets to correct inaccuracies, remix with other datasets to augment the data, or perform calculations on the data), *generally* improves data quality, for example, through reconciling entities and flagging amounts that are obviously incorrect. + +**FACT 3:** Data wrangling can also *introduce* errors if not tackled correctly. + +Crucial to ensuring the use of this data in articles or ensuring re-use by concerned citizens is the ability to show that the data is valid. In addition, maintaining a good relationship with public bodies who are confident that they are not being misrepresented in the data is vital to ensuring the data continues to be released in the first place. In practice, this means that the provenance of the data has to be clear including: + + * where the data originally came from (preferably a URL) + * whether anyone (e.g., government, community data wrangler, or OpenSpending) has worked on the data since it was published, and what steps they took to change the data (i.e., these steps should be reproducible to produce the same result) + +The OpenSpending team has gone to lengths to retain enough information to say who was responsible for both of the above. + +OpenSpending is a system, somewhat like a wiki, which allows you to track back through the data wrangling process and work out what changes were made to the data, when and by whom. + +## Error reporting in practice + +OpenSpending recently received a pointed inquiry from the U.K. Treasury disputing the claims we were making about the payment of British public money to a private company. Believing that an error had been introduced, we attempted to retrace our steps and find out where this had occurred, and who was responsible. + +As we discovered, the payment *had* actually taken place, but the the OpenSpending descriptions used to label the transaction were not sufficiently detailed to accurately reflect the item in question. + +With Spending Stories, we were able to retrace our steps because we had preserved a copy of the software tools we used for collecting the data (the data is published by about 50 public bodies, and must be downloaded, stitched together, and firmly molded into shape). These tools had been also made available to the public, so the Treasury and other concerned citizens could have checked our work themselves; the availability of this kind of check keeps all participants in the fiscal debate honest. + +What had gone wrong was a problem of terminology: The transactions existed, but ambiguous language had been used to describe them, glossing over the distinction between the government department reporting what money had been spent and the government agency which actually spent the money. The bodies in question were the Department of Health and a regional health care trust; this distinction is certainly one which a concerned citizen would expect to be made clearly -- so we should make sure our system makes it easy to know which question is being asked. + +## Checkpoints in OpenSpending + +In the short term, we are mitigating the problem of data errors as follows: + + * **Data provenance** - is the source identifiable and the process reproducible? OpenSpending encourages people to add modified datasets to a "package" in the Data Hub. This allows other users to see the original document alongside any modified documents and track the chain of changes made to see clearly which points errors could have been introduced. + * **Crowdsourcing feedback** on spending data. + * **Permitting re-use of the structured data** we present, so that it can inform decisions in other fact-checking systems. + +Ultimately, we will build our part of the ecosystem to provide feedback to the political process, by improving democratic discourse about the public finances. + +*Lucy Chambers is a community coordinator at the Open Knowledge Foundation. She works on the OKF's OpenSpending project and coordinates the data-driven-journalism activities of the foundation, including running training sessions and helping to streamline the production of a collaboratively written handbook for data journalists.* + +*Martin Keegan is a software engineer and linguist, currently leading the Open Knowledge Foundation's OpenSpending project. He is also on the Open Knowledge Foundation's board, and has worked for SRI, Citrix, University of Cambridge and co-founded and worked for various civil society organizations.* diff --git a/examples/openspending/content/blog/2011-12-12-data-seized-sanitised-and-sanity-checked-open-data-day-2011.md b/examples/openspending/content/blog/2011-12-12-data-seized-sanitised-and-sanity-checked-open-data-day-2011.md new file mode 100644 index 00000000..6c860137 --- /dev/null +++ b/examples/openspending/content/blog/2011-12-12-data-seized-sanitised-and-sanity-checked-open-data-day-2011.md @@ -0,0 +1,89 @@ +--- +authors: +- mark +redirect_from: /2011/12/data-seized-sanitised-and-sanity-checked-open-data-day-2011/ +title: Data = Seized, Sanitised and Sanity-checked. Open Data Day 2011 +tags: +- CKAN +- events +- IATI +- Open Data Day +- Publish What You Fund +--- +**This post is by Mark Brough, Research Officer at [Publish What You Fund](http://www.publishwhatyoufund.org/), [Lucy Chambers](http://okfn.org/members/lucychambers), Community Coordinator for OpenSpending, and [Irina Bolychevsky](http://okfn.org/members/shevski), Product Owner for CKAN. It is cross-posted on the [OpenSpending Blog](http://blog.openspending.org/2011/12/10/data-seized-sanitised-and-sanity-checked-open-data-day-2011) and the [Open Knowledge Foundation Blog](http://blog.okfn.org/2011/12/12/data-seized-sanitised-and-sanity-checked-open-data-day-2011) and Mark Brough's contribution is also featured on [aidinfolabs.org](http://www.aidinfolabs.org/archives/786).** + +**Saturday, December 3rd was Open Data Day, and London took the challenge to throw a hackday to help data be opened, cleaned and shown off to the world...** + +Fuelled only by enthusiasm, caffeine and 5 packets of ready-made popcorn, the CKAN, OpenSpending and IATI teams, along with some new faces, joined forces to liberate as much data as they could... + + + +## OpenSpending + IATI + CKAN + +As part of the IATI Open Data Day challenges, Mark Brough did some work to get the existing IATI Data into OpenSpending. David Read, from the CKAN team, and a new face to the data wrangling crew, Johannes, scraped data on aid donations from France and Austria that were locked-up in web apps in order to help fill in the gaps in the global aid data jigsaw puzzle. You can see the results on OpenSpending. + +* France: and on OpenSpending: +* Austria: , on OpenSpending: + +The French (AFD) and Austrian (ADA) aid data appears to be incomplete: the AFD's [2010 Annual Report] suggests that South Africa is the biggest recipient country, receiving €403 million, but in the data, Morocco is the biggest recipient and there are no transactions in South Africa. + +The Austrian Development Agency data was carefully cleaned by Johannes, with region and country codes being added for all entries to create a tidier dataset. However, the original data contained, for example, four different spellings of Bosnia and Herzegovina, suggesting that countries are being manually entered rather than selected from an existing list. [For 2010], the second biggest recipient of the Austrian Development Agency's aid (after aid not going to a specific country) appears to be Austria. + +Nevertheless, despite the issues surrounding data quality, it was a useful exercise to show both the value of open data - that if you release your data, you can do pretty cool things with it - and the costs of keeping it locked away, namely that the data then has to be scraped from sites in quite a labour-intensive way. + +These, along with many other datasets discovered on the day via tweets and emails have been added to the [Open Data Day Group](http://thedatahub.org/group/open-data-day) on [theDataHub.org](http://thedatahub.org). + +On the same day, we worked to get the data released as part of the International Aid Transparency Initiative into OpenSpending. You can see the results of the IATI wrangling process on [OpenSpending.org/iati](http://www.openspending.org/iati). This following section is written by Mark. + +### 1. Getting the data + +Downloading the existing IATI data has already become quite a big task; with 19 publishers so far, the data currently amounts to over 750MB with 1169 packages. Fortunately this is made easier by the IATI Registry, which provides an API to access all existing datasets, and a simple script (links at end) can retrieve all of the data. + +### 2. Extracting the data +Extracting the data from the XML files is more complicated. Although IATI data uses a standard schema, there are a few cases where publishers have either used the markup incorrectly, or else interpreted the definitions slightly differently. This can be simple problems such as stating that an organisation is “implementing” rather than “Implementing”, or placing the date within the text of the tag and not the “iso-date” attribute of that tag, or more significant problems such as placing implementing organisations in the “accountable” organisation field. + +However, these problems are still fairly limited and follow fairly regular patterns, so they are not too hard to overcome. There are more significant problems when some donors have for example used three-letter (ISO-3) country codes, rather than two-letter (ISO-2) country codes. (This is considered below in “next steps”.) + +### 3. Wrangling the data +OpenSpending is designed to show spending data, and has a powerful aggregation system to show large collections of transactions in a meaningful way. However, IATI data is organised by activities, with transactions nested within activities (projects), and – reflecting the business models of funders – activities sit within other activities (e.g., projects within programs), although they are not nested in the actual XML. Furthermore, one of the significant advantages of IATI compared to other aid data formats is that it permits multiple sectoral classifications, allowing you to assign a proportion of the value of an activity to each sector. So, you might have an activity that is 50% related to health and 50% to education. + +To prepare the data for OpenSpending, each transaction inherits the properties of its activity (and, if that activity has a parent, that parent activity’s title and description). Then, the transaction is broken out into mini transactions, with the proportion of the activity assigned to each sector used to assign a proportion of the value of the transaction to each sector. So, from transactions, you get mini “sector-transactions”. + +This takes about 40 minutes to compile, and then one final step remains: to convert the currencies to a single currency. Currently, USD, EUR and GBP amounts are used in the IATI data. All data is converted to USD using the average for 2010 from the OECD’s Financial Indicators (MEI) dataset. (This is also considered below in “next steps”.) + + +### 4. Loading the data +OpenSpending’s new web-based loading interface makes it relatively easy to load data in, although you currently also have to write a model and views (links at end). + + +### Results +The results can be viewed in the OpenSpending IATI dataset. You can explore the data by recipient country, sectors, funding organisation, and drill down through the data to see the data for an individual country. + + + +### Problems with the data +So far I’ve noticed the following problems: + +* “Unknown” recipient location is incorrectly marked as “South Sudan” +* Recipient countries are listed twice, as Spain has used ISO3 rather than ISO2 country codes. +* Sweden is listed as “Ministry of Foreign Affairs” (this is how they have listed themselves as the Funding Organisation in the data) +* Sweden’s implementing organisations have been lost as they placed them in the accountable organisation field. + +Please let me know if you see anything else problematic, if you have and criticisms of feedback of the way the data has been presented, or if you think there are other ways you’d like to be able to explore the data, based on the available dimensions. + +### Next steps +As mentioned above, there are some problems with the data which should properly be dealt with at the level of the donor agency. But there are others that will probably have to be dealt with by users of the data: + +* Mapping between different sector vocabularies, so that you can see all “Health” projects, and not only the health projects according to a single vocabulary +* Mapping between countries and regions, so that every project in a country has a related region +* Correctly converting currencies using the “value-date” column to get a more precise (at least month-specific) conversion. + +**What else have you noticed with the data? Is there anything else that should be changed? Anything interesting?** + +You can contact Mark about this data via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending) + +### Useful Links +* [IATI on OpenSpending](http://www.openspending.org/iati) +* [Data wrangling scripts and tools](https://github.com/okfn/iatitools) +* [Mapping spreadsheets](https://github.com/okfn/iatitools/tree/master/mapping) +* [Etherpad from Open Data Day - attendees and projects they worked on](http://ckan.okfnpad.org/opendataday) diff --git a/examples/openspending/content/blog/2012-01-12-civil-society-and-spending-data-who-is-mapping-the-money.md b/examples/openspending/content/blog/2012-01-12-civil-society-and-spending-data-who-is-mapping-the-money.md new file mode 100644 index 00000000..255b55b0 --- /dev/null +++ b/examples/openspending/content/blog/2012-01-12-civil-society-and-spending-data-who-is-mapping-the-money.md @@ -0,0 +1,54 @@ +--- +authors: +- lucy +redirect_from: /2012/01/civil-society-and-spending-data-who-is-mapping-the-money/ +title: "Civil Society and Spending Data: Who is mapping the money?" +tags: +- Contribute +- OSF +--- +**This post is by [Lucy Chambers](http://okfn.org/members/lucychambers), Community Coordinator on the OpenSpending project at the Open Knowledge Foundation. The post is cross-posted on the [Open Knowledge Foundation blog](http://blog.okfn.org/2012/01/12/civil-society-and-spending-data-who-is-mapping-the-money/).** + +We're excited to announce that, thanks to the generous support of the Open Society Foundations, OKFN's activities around financial transparency will expand to include a second pillar: next to the OpenSpending platform, we have just started a 6 month project to map the technology needs of Civil Society Organisations in relation to public spending and budget information. + +## We're going to be working on... + +* **Identifying CSOs around the world who are interested in working with spending data** - building on the existing network of contacts from the OpenSpending.org project. + +* **Connecting these CSOs with each other**, with open data communities and with other key stakeholders to exchange knowledge, experiences and best practices in relation to spending data + +* **Establishing how CSOs currently work with spending data**, how they would like to use it, and what they would like to achieve - including: + 1. what existing tools are being used + 2. what current technical needs are unmet + 3. what would be required to meet these needs and how feasible is it to tackle them + +* **Creating a registry of spending datasets**, from official and unofficial sources in theDataHub.org +* **A Spending Data Manual** - A wiki-like, community driven manual on acquiring, working with, publishing and archiving spending data, based on input and exchanges with CSOs we talk to.This will augment and reference existing publications from numerous organisations as well as channelling the results of our research into two areas: + * **A section to help CSO’s clarify their demands towards governments:** e.g. guidance on open licensing and structured data formats, applicable for spending data. + * **A section focused on best practice for CSOs when using and reusing spending data:** for example collaborative processes such as data-sharing. +  +* **Running Spending Analysis Sessions with CSOs**, both in person and virtually. We’re interested in learning from about what data people are trying to acquire / having difficulty in doing so, how they plan to use the data to further their mission and learning what barriers, legal, technical and otherwise could be removed to make their jobs easier. + + +* **Getting Spending Data from numerous countries loaded into OpenSpending.org** - with the support of CSOs, OKFN developers, and volunteers from the open data community. We we're interested in are using the OpenSpending.org tools, and collect input from them on how these could be improved to meet their needs. + + + +## Vision: Improved Spending Data Literacy, Sharing and Re-use amongst CSOs around the world + +We are very keen to help more groups and individuals around the world to use and work with spending data more effectively to do the things they care about - whether this is investigative journalism, evidence based policy-making, political campaigning, budgeting or creating new useful applications and services. + +In particular, we would like to document and spread best practices in the legal and technical aspects of reusing public information, and enabling re-use and better collaboration around this material. + +### Ultimately we would like to: +* **Build stronger, broader communities** of groups and individuals who work together to acquire, use, and openly share spending data +* **Increase ‘literacy’ around spending data** - enabling more CSOs to understand and work with large and complex spending datasets to help them to pursue their objectives +* **Encourage more CSOs to publish datasets which they acquire**, use or create in machine readable formats, under open licenses, to avoid duplication of effort and enable CSOs to build on each others’ work, to harness external expertise more effectively and to facilitate stronger collaboration between different organisations who are interested in spending information + +## How can I get involved? + +* **Join the Working Group on Spending Data**. The working group will bring together data experts and CSOs who will help to weave a community of best practice around spending data, collect and provide feedback on material for the manual and help to develop the network of those collaborating around and sharing spending data. More details about the working group can be found on this [wiki page](http://wiki.openspending.org/Working_Group). + +* **Write for the [Spending Data Blog](http://blog.openspending.org)** - we're interested in posts by and about CSOs who work with spending data, observations on the current status quo on releasing data in your area. Anything from short comment pieces to full proposals for what could be done, legal, technical or otherwise, to improve the situation in the sphere where you work. Contact details as above. + +**If you would like to get started, or know of organisations we should extend the invitation to: drop us an email via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending) or contact me directly via info [at] openspending.org. ** diff --git a/examples/openspending/content/blog/2012-01-17-open-bookkeeping-what-role-can-accountants-play-in-open-spending-budget-projects.md b/examples/openspending/content/blog/2012-01-17-open-bookkeeping-what-role-can-accountants-play-in-open-spending-budget-projects.md new file mode 100644 index 00000000..b3a59d7a --- /dev/null +++ b/examples/openspending/content/blog/2012-01-17-open-bookkeeping-what-role-can-accountants-play-in-open-spending-budget-projects.md @@ -0,0 +1,22 @@ +--- +authors: +- lucy +redirect_from: /2012/01/open-bookkeeping-what-role-can-accountants-play-in-open-spending-budget-projects/ +title: "Open Bookkeeping: What role can accountants play in Open Spending & Budget Projects?" +--- +The next OpenSpending online community meeting will take place on *Thursday, 19th Jan - 6pm GMT*. + +## The topic + +Accountants spend their entire working lives mapping the money: + +* How can their expertise be put to good use in Open Spending Data projects? +* What interesting initiatives are going on around the world which could benefit from the input of accountants? + +Via OpenClipArt + +All are welcome! If you'd like to contribute to the discussion by joining the call, please just add your name and Skype ID to [the pad](http://wdmmg.okfnpad.org/community-2012-01-12) + +Please feel free to share with colleagues friends and other communities. + +N.B. Over the next weeks, we'll be trying to theme the discussions and proactively invite people along to join them. If you have a suggestion for a topic you think the group should discuss, please drop us a line via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2012-01-23-transparency-and-technology-in-brazil-linking-politicians-to-bad-entrepreneurs.md b/examples/openspending/content/blog/2012-01-23-transparency-and-technology-in-brazil-linking-politicians-to-bad-entrepreneurs.md new file mode 100644 index 00000000..c1828a67 --- /dev/null +++ b/examples/openspending/content/blog/2012-01-23-transparency-and-technology-in-brazil-linking-politicians-to-bad-entrepreneurs.md @@ -0,0 +1,60 @@ +--- +authors: +- fabiano +redirect_from: /2012/01/transparency-and-technology-in-brazil-linking-politicians-to-bad-entrepreneurs/ +title: "Transparency and technology in Brazil: linking politicians to bad entrepreneurs" +tags: +- Data Journalism +- Spending Stories +--- +**This story by Fabiano Angélico, who formerly worked at Transparencia Brasil, is about how technology and the help of coders can be used to highlight links between politicians and corrupt entrepreneurs. It is followed by a brief "Behind the News" interview which shows some of the time costs of datawrangling and problems faced when getting the story out.** + +How can transparency and technology point out connections between politicians and bad entrepreneurs? Well, first of all you will need some information about the politicians and about the entrepreneurs. + +In Brazil, in spite of the historical lack of transparency in governments (Brazil's freedom of information law was sanctioned just late last year), the Electoral Court has been proactively providing information on political candidates since 2002. One piece of info is the financial donation to the candidates, containing info about who is donating to whom and how much. Although this database is released only after the elections -- the info would surely be more powerful if it were released DURING the political campaigns --, one must admit this is a rich source of information. + + + +January, 2010. Elections for President and for the Parliament, as well as for State Governors and State Parliaments, would happen in only 9 months time, in October. However, many people were already discussing them. + +At that time, 2010 had just begun, I was at work, thinking of how to find rich and useful information on the candidates. Then I was reminded of the so-called ["Dirty List"](http://www.mte.gov.br/sgcnoticia.asp?IdConteudoNoticia=6680&PalavraChave=lista%20suja) -- this is a list regularly published by the Ministry of Labour which indicates the companies and farmers who are caught by government officials using workers in very lousy conditions, similar to slavery. + +The list published in the Ministry's website is in not-so-friendly PDF format, but it has a plus: there is not only the name of the companies or the entrepreneur/farmer, but also their registry numbers within the government. I remembered that in the Electoral Court one can also find the numbers. That was important because having the registry numbers would avoid ambiguities. + +I had both lists: the donators to the previous elections (2008, 2006, 2004 and 2002) and the "Dirty" companies. But I had a problem; I did not know how to matchup the datasets. My tech knowledge allowed me to transform the PDFs into CSV, but I could no go further without help. + +I then sent the datasets, in CSV format, to [Transparencia Hacker](http://thacker.com.br/), a Google Groups list which now gathers over 800 people interested in the connections between transparency and politics/public administration. + +Within 2 days, the guys made the datasets talk, and we found that 16 politicians had been elected with the help of "Dirty" money in the 4 previous elections. Other 13 politicians had received donations from the "Dirty List" but had not succeeded in winning the elections. + +[A local newspaper told the story](http://www.agenciasebrae.com.br/noticia.kmf?canal=36&cod=9376495&indice=0). + +In October 2012, there are local elections in Brazil. Hope we can shed even more light in the candidates. + +# Behind the news: + +## Roughly how long did it take you to extract the data from the PDFs? Do you know how long the guys from Transparencia Hacker spent working on the data? + +This was kind of easy. It took me just some minutes. The "Dirty List" is a 20-page PDF. I always use a website to convert it into xls or csv (I like [Cometdocs](http://www.cometdocs.com/) for this work). + +[Here](http://portal.mte.gov.br/data/files/8A7C812D3374524E0133835496AF7D72/CADASTRO%20DE%20EMPREGADORES%2008%20de%20novembro%202011.pdf) is the Dirty List, in PDF (last updated on the 8th of November, 2011; the list we used is in CSV but it it very outdated because it was due to January 2010) +Here are the Electoral Court pages for the list of donators: [2002](http://www.tse.jus.br/internet/eleicoes/2002/prest_blank.htm), [2004](http://www.tse.jus.br/internet/eleicoes/2004/prest_blank.htm), [2006](http://www.tse.jus.br/eleicoes/eleicoes-anteriores/eleicoes-2006/prestacao-de-contas-eleicoes-2006), [2008](http://www.tse.jus.br/eleicoes/contas-eleitorais/candidatos-e-comites/prestacao-de-contas-eleitorais-2008) and [2010](http://spce2010.tse.jus.br/spceweb.consulta.prestacaoconta2010/pesquisaCandidato.jsp). + +What I asked the Transparencia Hacker community was to check whether the CNPJs (companies register number within the governments) in the CSV would match any item in the Electoral Court webpage. The guys worked on the data for 2 days. + +## Is sufficient data available to visualise the total amount lobbyists donated to political campaigns, and would it be useful to / no? If you were to visualise the info - what would the priorities be to show? Would any tools be useful to explore the data? + +Yes, there is enough data. And YES, it would be very useful to visualize those links. I would prioritise the presidential and governor candidates as well as some Congressmen who hold top-positions in both Houses of Congress. Also, the donations to political parties (not to individual politicians) would be a plus. + +A search form would be very useful. The search could have filters for position (Presidential candidate, governor candidate, political party etc), geography (Brazil, states) and donators (with no filters, just a blank for writing) + +## In your ideal world, in time for the impending elections - what would be done differently from last time? Any additional data you would like to see released? + +I'd have to think more carefully to respond that, but concerning additional data: the number which identifies the market (the field) in which the companies work. + +*Interested in writing a "Behind the News" piece for the OpenSpending blog? Get in touch via our [twitter account](https://twitter.com/#!/openspending) or email **info [at] openspending.org**.* + +Some useful links (mainly in Portuguese): + + * [Brasil adopts access to information law](http://www.article19.org/resources.php/resource/2862/en/brazil-adopts-access-to-information-law) + * [View the Dirty List in Full](http://www.mte.gov.br/sgcnoticia.asp?IdConteudoNoticia=6680&PalavraChave=lista%20suja) diff --git a/examples/openspending/content/blog/2012-01-23-updates-from-the-openspending-dev-team.md b/examples/openspending/content/blog/2012-01-23-updates-from-the-openspending-dev-team.md new file mode 100644 index 00000000..88138629 --- /dev/null +++ b/examples/openspending/content/blog/2012-01-23-updates-from-the-openspending-dev-team.md @@ -0,0 +1,16 @@ +--- +authors: +- friedrich +redirect_from: /2012/01/updates-from-the-openspending-dev-team/ +title: Updates from the OpenSpending Dev Team +tags: +- Releases +- Spending Stories +- Updates +--- +# What are we focusing on this week? + +* Working on implementing [Collections](http://wdmmg.okfnpad.org/collections) <- draft notes, beware. +* Prototyping the Compar-o-tron [Mockup 1](http://www.flickr.com/photos/okfn/4623584907/), [Mockup 2](http://www.flickr.com/photos/okfn/4624190848/). +* Continuing work on Embeddable widgets for Spending Stories. + diff --git a/examples/openspending/content/blog/2012-01-30-hakuna-my-data-nbo-data-bootcamp.md b/examples/openspending/content/blog/2012-01-30-hakuna-my-data-nbo-data-bootcamp.md new file mode 100644 index 00000000..76d7a0b7 --- /dev/null +++ b/examples/openspending/content/blog/2012-01-30-hakuna-my-data-nbo-data-bootcamp.md @@ -0,0 +1,34 @@ +--- +authors: +- friedrich +redirect_from: /2012/01/hakuna-my-data-nbo-data-bootcamp/ +title: "Hakuna My Data: NBO Data Bootcamp" +tags: +- Data Journalism +- events +- Kenya +- NBO +- x +--- +**This post is by [Friedrich Lindenberg](http://okfn.org/members/pudo), developer on OpenSpending.** + +>"My Name is XXXX, I am a member of the Kenyan parliament for the constituency of XXXX in the 2007-2012 election cycle. During my time in parliament, I have positioned myself against taxes for MPs. + +>Of the Development Funds allocated to my constituency, I have spent 12mn KSH in 2010 and 8mn KSH in 2009. Since 2007, I've funded 201 projects, of which 72 (9mn KSH) related to Education, 56 (7.2mn KSH) related to Health and 20 (4.2mn KSH) to Infrastructure. + +>The largest projects I have funded include... " + +Auto-generated, spending data-driven campaign speeches like this are just one of the many ideas of the Data Bootcamp that took place in Nairobi last week. Invited by the African Media Initiative and the World Bank Insititute, about 70 participants - both journalists and developers - met on Strathmore University's campus to learn and practise both the skills and tools required for data-driven reporting. + +The four-day programme combined tools training with practical work in small groups. Elena Egawhary (BBC NewsNight) gave a workshop on data analysis in Excel, Sreeram Balakrishnan (Google Fusion Tables) introduced both Refine and Fusion Tables. Team members from both the Kenya data portal and the World Bank finance site presented their respective offerings, while Gregor and myself from the OpenSpending team gave intros to web scraping and advanced +map visualisation. + + + +During group work, journalists and developers teamed up to try their newly learned skills in different domains ranging from sports (football player profiles) to education (missing toilets in schools, "The Shit Ordeal") and the financial transparency story-telling mentioned above. + +The workshop also served as a community-building event for Kenya's young and impressive Open Data initiative. Future events, aimed at civil society organisations and polictical actors will help to further promote the re-use of government information released through the initiative. + +All this is happening in a place where transparency is an essential tool to be developed: Not only is the access to information now guaranteed by the 2010 Kenyan constitution, there are also major political issues that deserve close attention from local and international watchdogs. These include not only the ongoing incursion of Kenyan troops into Somalia in an effort to fight Al-Shebab terrorist groups, but also the upcoming nationwide elections in December 2012. The elections will instate a new bicameral system of government, with many previously unknown candidates standing for office. In the previous 2007 vote, bad polling station data had quite literally led to widespread unrest and thousands of deaths across the nation. + +In all, it was a fantastic to get in touch with the Kenyan participants of the workshop and to see how the organizers of the event - a brilliant team including Craig Hammer, Justin Arenstein and Jay Bhalla - are working to foster an open data community in this bustling developing nation.Given the great ideas generated during the team sessions, I'm sure this work will soon bear its first fruits. diff --git a/examples/openspending/content/blog/2012-02-15-the-czech-budget-on-line-the-half-success-story.md b/examples/openspending/content/blog/2012-02-15-the-czech-budget-on-line-the-half-success-story.md new file mode 100644 index 00000000..5c7b40e9 --- /dev/null +++ b/examples/openspending/content/blog/2012-02-15-the-czech-budget-on-line-the-half-success-story.md @@ -0,0 +1,37 @@ +--- +authors: +- michal +redirect_from: /2012/02/the-czech-budget-on-line-the-half-success-story/ +title: "The Czech budget on-line: the half success story" +tags: +- Working Group +--- +**This post is by Michal Škop, of KohoVolit.eu.** + +The half-success story of implementing OpenSpending.org and OtwarteDane.pl into BudovaniStatu.cz ('Building of the State', the name referes to Peroutka's book) + + +It all started almost 2 years ago. Our partner NGO NasiPolitici.cz started to think about putting the Czech public money data on the web and asked us at KohoVolit.eu if we were interested. And we said yes, we always wanted to do something 'about money' (we used to be a parliamentary watchdog only till then). + +We found out that there is a huge amount of public financial data available on-line. Every single public organization has to fill several detailed accounting forms every year, the oldest data are from 1994 (not published, but they are there). And it is available even in xml. Can you ask for more? + +Later on, we found that there were some serious catches. The Ministry of finance, which provided the data, severely limited the number of downloads from one IP. It would have taken us a couple of months just to download everything (some 60 GB of data). The Tor and mobile connection (changing IP) came in useful. The forms were in xml, but mixing raw basic data with sums with no clear distinction between them at all. Funny. They changed the system for 2010. Et cetera. We were progressing rather slowly, with no financial support at all. + + + +Finally, help from Anticorruption Endowment came and we got funding for about two month (developer) to build a site connecting (just) the government budgets with the politicians. That was important, I could not just show the data in some nice way, I needed to do other things with the application – showing historical data, connecting to politicians. + +I spent a month just fiddling with the data, trying to find a suitable +a) data storage and +b) application to build on. + +I tried OpenSpending.org first, but I was not able to set up the data there. I tried to tweak our parliamentary API, but it was just too much work, I would not be able to finish it in time. After a few weeks, I still was not sure if I would get the results using OpenSpending.org. The guys behind OtwarteDane.pl were very helpful and so we decided to store the data with them. + +I did not use OpenSpending.org's API, but their bubbletree chart was good. I needed to catch a few bugs, but it took me just a few days to get it running more-or-less in a way I wanted (well yes, I still need to clean the code for 'pull request'). And – importantly – it was possible to build our application(s) on it. + +I think, we have hit the bubbletree's limit on number of bubbles there. It runs rather well with data we limited it to later (about 3600 bubbles), but it takes javascript about 10 sec on my medium computer to process the full data, 24000 bubbles for 2010 year, Opera cannot handle it and IE had problems, too (try it on our development site). + +And how about the 'where does my taxes go' app? Well, it was rather easy from the developer's view. I could copy the British idea, just program it in Javascript instead of the Flash. The hard part was the economics here. We could not use just the income tax as it accounts for about 10 % of all the taxes only (the VAT, the health tax, the social tax are more important). The taxes are messy. The general financial reporting is a mess, too. I have found about 15 % difference in 'public taxes' in different financial reports from Czech Statistical Office. So which one to use to calculate the overall taxes? But this is just one reason more why OpenSpending.org will be useful, to standardize this mess. + + +For the future, we will update the project once the 2011 data is available. We shall solve the problem with bubbles' scaling. We will write analyses based on it mainly push others to do it. And I already have the Prague 2012 budget data ready to bubble... diff --git a/examples/openspending/content/blog/2012-02-16-thekit.md b/examples/openspending/content/blog/2012-02-16-thekit.md new file mode 100644 index 00000000..d3ac8ced --- /dev/null +++ b/examples/openspending/content/blog/2012-02-16-thekit.md @@ -0,0 +1,26 @@ +--- +authors: +- friedrich +redirect_from: /2012/02/thekit/ +title: Announcing the Where Does My Money Go? Assembly Kit +tags: +- Releases +- Updates +--- +Over the past few months, we've made a lot of progress on OpenSpending. The core of the application is now mostly stable and it is getting ever easier to load data into the platform through the web-based dataset editor. Yet, inevitably, this raises a simple question: I've imported my data, what next? + +Thanks to our [API](http://openspending.org/help/api.html), there can be an infinite number of answers. With the [BubbleTree](http://okfnlabs.org/bubbletree/) diagram, the [Daily Bread](http://wheredoesmymoneygo.org/dailybread.html) application and the transactional spending browser, we have a few simple answers. + +But as [Michal Škop blogged recently](http://blog.openspending.org/2012/02/15/the-czech-budget-on-line-the-half-success-story/), up to now it has been fairly difficult to use both these widgets and the OpenSpending API to create custom front-ends. + +To make things easier, we've now created the [Assembly Kit](https://github.com/openspending/wheredoesmymoneygo.org). The kit is in fact the source for a newly styled version of the [Where Does My Money Go?](http://wheredoesmymoneygo.org/) site that has gone live yesterday. [Contained in this](https://github.com/openspending/wheredoesmymoneygo.org) is a clean set of templates that can anyone who knows basic HTML can easily use to make a lightweight, white-label budget visualization site, styled according to your own wishes. + +Screen Shot 2012-02-16 at 2.35.48 PM + +A set of widgets are included and can be adapted to another dataset with just a few edits. And since everything runs against the OpenSpending API, you don't need to run your own database. Instead, you can [load your data into OpenSpending.org](http://wiki.openspending.org/Loading_into_OpenSpending) and then customise the user facing side - for example, you can just use a generic blog or a set of static HTML files. + +Our next step in March will be to make it easier for users - especially Journalists - to create custom configurations for the visualizations via a graphical interface, save specific views and share them through a simple embed code. We'll also work to roll out the mapping support more widely and to create more custom apps on top of the API. + +Our goal is to make OpenSpending the easiest way to publish and analyze a government finance dataset - with your help! So please provide us with feedback and contribute your own visualizations to the OpenSpending platform. + +* [Assembly Kit](https://github.com/openspending/satellite-template) diff --git a/examples/openspending/content/blog/2012-02-24-how-spending-stories-fact-checks-big-brother-the-wiretappers-ball.md b/examples/openspending/content/blog/2012-02-24-how-spending-stories-fact-checks-big-brother-the-wiretappers-ball.md new file mode 100644 index 00000000..9c712633 --- /dev/null +++ b/examples/openspending/content/blog/2012-02-24-how-spending-stories-fact-checks-big-brother-the-wiretappers-ball.md @@ -0,0 +1,56 @@ +--- +authors: +- lucy +redirect_from: /2012/02/how-spending-stories-fact-checks-big-brother-the-wiretappers-ball/ +title: How Spending Stories Fact Checks Big Brother, the Wiretappers' Ball +tags: +- big brother +- Data Journalism +- pbs +- privacy international +- Spending Stories +- spending stories +- surveillance +--- +**This piece was co-written with Eric King of [Privacy International](https://www.privacyinternational.org/) and comes as Privacy International launches a huge new data release about companies selling surveillance technologies. It is cross-posted on the [MediaShift PBS IDEA LAB](http://www.pbs.org/idealab/)** + +Today, the global surveillance industry is estimated at around $5 billion a year. But which companies are selling? Which governments are buying? And why should we care? + +We show how the [OpenSpending platform](http://openspending.org/) can be used to speed up fact checking, showing which of these companies have government contracts, and, most interestingly, with which departments... + +## The Background + +Big Brother is now indisputably big business, yet until recently the international trade in surveillance technologies remained largely under the radar of regulators and civil society. Buyers and suppliers meet, mingle and transact at secretive trade conferences around the world, and the details of their dealings are often shielded from public scrutiny by the ubiquitous defence of 'national security'. Perhaps unsurprisingly, this environment has bred a widespread disregard for ethics and a culture in which the single-minded pursuit of profit is commonplace. + + + +For years, European and American companies have been quietly selling surveillance equipment and software to dictatorships across the Middle East and North Africa - products that have allowed these regimes to maintain a stranglehold over free expression, smother the flames of political dissent and target individuals for arrest, torture and execution. + +They include devices that intercept mobile phone calls and text messages in real time on a mass scale, malware and spyware that gives the purchaser complete control over a target's computer and trojans that allow the camera and microphone on a laptop or mobile phone to be remotely switched on and operated. These technologies are also being bought by Western law enforcement, including small police departments in which the ability of officers to understand the legal parameters, levels of accuracy and limits of acceptability is highly questionable. + +The data that has just been released on the [Privacy International Website](https://www.privacyinternational.org/big-brother-incorporated/countries) included the following: + + 1. An updated list of companies selling surveillance technology, and + 2. Naming all the government agencies attending an international surveillance trade show known as the wiretappers ball. + +Some names are predictable enough: [the FBI](https://www.privacyinternational.org/big-brother-incorporated/countries/United%20States/US_Federal_Bureau_of_Investigation_FBI_-_OTD), the [US Drug Enforcement Administration](https://www.privacyinternational.org/big-brother-incorporated/countries/United%20States/US_Drug_Enforcement_Administration_DEA_-_ONSI), the [UK Serious Organized Crime Agency](https://www.privacyinternational.org/big-brother-incorporated/countries/United%20Kingdom/UK_Serious_Organised_Crime_Agency_SOCA_) and [Interpol](https://www.privacyinternational.org/big-brother-incorporated/countries/International/Interpol), for example. The presence of others is deeply disturbing: the national security agencies of [Bahrain](https://www.privacyinternational.org/big-brother-incorporated/countries/Bahrain/Bahrain_National_Security_Agency) and [Yemen](https://www.privacyinternational.org/big-brother-incorporated/countries/Yemen/Yemen_National_Security_Agency), the embassies of [Belarus](https://www.privacyinternational.org/big-brother-incorporated/countries/Belarus/Belarus_Embassy) and the [Democratic Republic of Congo](https://www.privacyinternational.org/big-brother-incorporated/countries/Belarus/Belarus_Embassy) and the [Kenyan intelligence agency](https://www.privacyinternational.org/big-brother-incorporated/countries/Kenya/Kenya_National_Security_Intelligence_Service), to name but a few. A few are downright baffling, like the [US department of Commerce](https://www.privacyinternational.org/big-brother-incorporated/countries/United%20States/US_Department_of_Commerce) or the [US Fish & Wildlife Service](https://www.privacyinternational.org/big-brother-incorporated/countries/United%20States/US_Fish_%2526_Wildlife_Service) and [Clark County School District Police Department](https://www.privacyinternational.org/big-brother-incorporated/countries/United%20States/Clark_County_School_District_Police_Department). + +Now, with the aid of OpenSpending, anyone can cross reference which contracts these companies hold with governments around the world. The investigation continues... + +## Using OpenSpending to speed up fact-checking + +Privacy International approached the Spending Stories team to ask for a search widget to be able to search across all of the government spending datasets for contracts held between governments and these companies (until this point, it had only been possible to search one database at a time). + +The Spending Browser is now live at . And, as the URLs correspond to the queries, individual searches can be passed on for further examination and, importantly, embedded in articles directly. [Try it yourself](http://openspending.org/) against the list of companies listed in [the Surveillance Section of the Privacy International Site](https://www.privacyinternational.org/big-brother-incorporated/countries) (Just enter a company e.g. 'Endace Accelerated' into the search bar). + +The Spending Browser will become increasingly more powerful as ever more data is loaded into the system. + +Want to help make this tool even more powerful? [Get involved](http://openspending.org/getinvolved) and help to build up the data bank. + +## Coverage + +You can read more about the background to these stories on the Privacy International Site and recent coverage by the International Media: + +* [Privacy International investigates the sale of surveillance technology](https://www.privacyinternational.org/big-brother-incorporated) +* Guardian [Surveillance trade shows: which government agencies attend?](http://www.guardian.co.uk/news/datablog/2012/feb/07/surveillance-shows-attendees-iss-world) +* Wall Street Journal [High-Tech Surveillance Comes to Small Towns](http://blogs.wsj.com/digits/2012/02/06/high-tech-surveillance-comes-to-small-towns/?KEYWORDS=privacy) diff --git a/examples/openspending/content/blog/2012-02-29-open-meeting-software-for-participatory-budgeting.md b/examples/openspending/content/blog/2012-02-29-open-meeting-software-for-participatory-budgeting.md new file mode 100644 index 00000000..f26be252 --- /dev/null +++ b/examples/openspending/content/blog/2012-02-29-open-meeting-software-for-participatory-budgeting.md @@ -0,0 +1,49 @@ +--- +authors: +- lucy +redirect_from: /2012/02/open-meeting-software-for-participatory-budgeting/ +title: "Open Meeting: Software for Participatory Budgeting" +tags: +- pb +- Updates +- Working Group +--- +There are already shining examples of direct forms of democracy and deliberation going on around the world but many of them are small scale, local and idiosyncratic solutions. Can technology help to take these discussions to the next level and offer templates for solutions that could be applied all over the world? + +* **When**: Today, 5pm GMT +* **How to join**: Add your name and Skype ID to the [etherpad](http://wdmmg.okfnpad.org/pb) + +We're also conducting a **Software for Participatory Budgeting Census** - if you know of examples which should be in there please add them [to this spreadsheet](https://docs.google.com/spreadsheet/ccc?key=0AvoV_cBqwo28dE9fZy02NEt2UGxPTnRQMTEzaUhTOGc#gid=0)! + +CC-BY St Peter's Community News + +Suggested topics (please feel free to add to these in the etherpad): + +* Personal experience with using software for PB: + * Any shining examples stand out? + * Any frustrations +* Case Studies + * Geographical focus (idiosyncrasies we need to take into account) + * Mobile + * Web +* Measures of success + * PB has many pieces, which software is best for which piece? + * Setting up a common framework for impact evaluation + * Scalability + * Penetration + * User retention + * Epistemic value +* Relevant topics for PB? + * deliberation on entire budgets at once + * local / national + * demand and supply of services & infrastructure + * problem-solving (a la fix my street - requests for more funds to be directed to solve a problem) + * allocating resources for area regeneration + * directing statutory funds to voluntary sector organisations + * Structural funds - national co-funding + * Exploring tradeoffs +* Which stages of the PB cycle to focus on? +* Probably no universal solutions - but what are the minimum common principles which can be identified and is there a tech solution for them? +* Low-hanging fruit - what are the next steps in coming months to take this to the next level? + +If you can't make the call - please feel free to contribute to the discussion via the [Participatory Budgeting Google Group](http://groups.google.com/group/participatorybudgeting?pli=1). diff --git a/examples/openspending/content/blog/2012-03-06-calculating-portugal's-taxes.md b/examples/openspending/content/blog/2012-03-06-calculating-portugal's-taxes.md new file mode 100644 index 00000000..85db105a --- /dev/null +++ b/examples/openspending/content/blog/2012-03-06-calculating-portugal's-taxes.md @@ -0,0 +1,21 @@ +--- +author: $authornamehere +redirect_from: /2012/03/calculating-portugal's-taxes/ +title: Calculating Portugal's taxes +tags: +- Contribute +- portugal +--- +**This post is by Nuno Moniz, who has recently developed an application to allow citizens in Portugal to calculate where their taxes go. Portuguese citizens can see how much they contribute to the State on a daily, weekly and monthly basis [here](http://www.nunomoniz.com/orcamento/) and if they live in the Azores Autonomous Region they will also be able to see how much they contribute at a local level [here](http://www.nunomoniz.com/orcamento/acores/).** + +During this last year or so, Portugal has been submerged in a vigorous discussion that concerns the economy, finances, social issues… but above all, money. The debt, the obligations, the budgets... + +Although in the latter years the Portuguese Government has shown some improvements in terms of e-democracy, mainly related to the public administration, when it comes to open data only now can we see some light at the end of the tunnel. The new Government Data Portal was launched in November, and it already has some data available. It presents a great opportunity for the open data community in Portugal to start joining efforts. Some projects had previously been developed, and I believe the most significant was http://demo.cratica.net, a parliament tracker. + +In the late October, when the first draft for the 2012 Portuguese State Budget was delivered to the Parliament, and inspired by a considerable number of interesting projects that I have been following regarding open data, I thought about developing a simple and different way to visualize what the Budget holds. Soon I found out that thanks to OKFN’s BubbleTree the work load could be really reduced. Great news and great help. + +That enabled me to launch a Portuguese 2012 Budget Visualization in a week. It took around two days only to extract the data from the budget document. This shows exactly the difficulty of understanding one of the most important State documents. In addition to most divulged projects of this sort, I added some additional information which that would be fun for people to see: the monthly, weekly and daily contributions to the various objects of the State Budget. It was very interesting to see the final result, so I continued and replicated the initiative to the Azores Autonomous Region of Portugal also. + +I’m currently working on my thesis regarding open legislation. By the summer we’ll have about three years of open Portuguese legislation available :) In the meantime, the Government Data Portal holds some data that would be very interesting to push the growth of the open data community in Portugal. And since no one seems to discuss anything else than economy in the news, I have also started working on the Public Contract and Direct Adjudications (celebration of contracts without public auction). Let’s see what it will show. + +Please send any feedback or questions you have for Nuno via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2012-03-06-participatory-budgeting-and-technology.md b/examples/openspending/content/blog/2012-03-06-participatory-budgeting-and-technology.md new file mode 100644 index 00000000..46e16f4f --- /dev/null +++ b/examples/openspending/content/blog/2012-03-06-participatory-budgeting-and-technology.md @@ -0,0 +1,50 @@ +--- +authors: +- lucy +redirect_from: /2012/03/participatory-budgeting-and-technology/ +title: Participatory Budgeting and Technology +tags: +- GIFT +- Working Group +--- +**Last week we held our first Open Meeting on [Software for Participatory Budgeting](http://blog.openspending.org/2012/02/29/open-meeting-software-for-participatory-budgeting/). We just scratched the surface of this huge topic, but here are the first notes. A full write up in blog-post form will follow after a few more calls** + +You can contribute to and edit these notes directly via this [wiki page](http://wiki.openspending.org/Meetups/Participatory_Budgeting) + +If you are interested in joining the next call, **Thursday 8th March, 5pm GMT**, please add your name and Skype ID to the [etherpad](http://wdmmg.okfnpad.org/pb2). In the meantime, discussion will continue via the [Participatory Budgeting Google Group](http://groups.google.com/group/participatorybudgeting?pli=1). + +## Outline of discussion (more detailed notes on the wiki) + +* We worked a lot around the PB budget software census - please continue to add examples and thoughts [here](http://bit.ly/y7uyEI) + +## Topics raised + +* Before examining technical approaches, what should we bear in mind? [Notes in Wiki section](http://wiki.openspending.org/Meetups/Participatory_Budgeting#Before_examining_technical_approaches.2C_what_should_we_bear_in_mind.3F) +* Personal experiences and lessons learned in using software for PB. [Notes in Wiki section](http://wiki.openspending.org/Meetups/Participatory_Budgeting#Personal_experiences_in_using_software_for_PB.) +* Classification of PB tools in the census. Please feel free to comment on and edit the classifications in the [wiki page](http://wiki.openspending.org/Meetups/Participatory_Budgeting): + * **Deliberation** - *Allows user input, crowdsourcing ideas and facilitating discussion e.g. collecting ideas for projects * + * **Simulation** - *Allows participants to explore how certain spending/revenue choices impact the budget. Such applications are usually educational.* + * **Cuts and Additions** *Users given simple choice to prioritise a given choice more or less highly. Can be used to propose politically unpopular activities (i.e. Poison List).* + * **Trade Offs** *Users given context on the impact of proposed choices and asked to assess relative benefits * + * **Personal Impact** *Users shown what the impact on them personally would be (e.g. increasing spending above budget levels increasing the amount of tax they pay as an individual)* + * **Problem Fixing** *Ask for allocations of more money to solve particular problems that people care about (e.g. on a Fix My Street model)* + * **Invite to offline-participation** *No survey conducted online, but technology used to issue invitations e.g. to town hall meetings where projects will be discussed* + * **Kickstart/Pledgebank** *If additional funds required to get a project off the ground - feedback and the opportunity for citizens to make in-kind or cash contributions* + * **Long Term** *e.g. capital investments* + * **Immediate Term** *e.g. for the next year's budget* +* Tools to aid comprehension of the budget process: How do we ensure that people casting their vote through this system genuinely understand the choices they are about to make? [Notes in Wiki section](http://wiki.openspending.org/Meetups/Participatory_Budgeting#Educational_tools.2C_how_to_better_aid_comprehension_of_the_budget_process) +* Who are the users? [Notes in Wiki Section](http://wiki.openspending.org/Meetups/Participatory_Budgeting#Who_are_the_users.3F) +Clear feeling amongst participants that any application that is built must address the needs of the entire spectrum of users. *I have added a key in a tab on the PB census called 'User Key' - where we briefly attempted to categorise the types of users who may be expected to interact with this system - I would be grateful to anyone who would like to add to it and also help me to complete the 'Types of user' column on the main sheet! * + +## Where Do We Go From Here? + +You decide, but I would suggest: + +* Planning a followup call (**5pm GMT, Thursday 8th March**, please add agenda items to the [etherpad](http://wdmmg.okfnpad.org/pb2)) + +And in the meanwhile: + +Please share your thoughts on the discussion (please add anything if I have forgotten) via the [Participatory Budgeting Google Group](http://groups.google.com/group/participatorybudgeting?pli=1): + + * Categorisations and + * User groups section of the census diff --git a/examples/openspending/content/blog/2012-03-08-uk-25k-spending-data.md b/examples/openspending/content/blog/2012-03-08-uk-25k-spending-data.md new file mode 100644 index 00000000..df5b7b98 --- /dev/null +++ b/examples/openspending/content/blog/2012-03-08-uk-25k-spending-data.md @@ -0,0 +1,89 @@ +--- +author: $authornamehere +redirect_from: /2012/03/uk-25k-spending-data/ +title: UK 25k Spending Data +--- +Since the question just came up about what's going on there, figure +this is a good day to tidy this up and send it out... + + +Most people who managed to submit sources sent in data that was +substantially correct, with the only inconveniences being: + + - arbitrary variation in spelling of column headers. Standardise this + - reporting of VAT: some include it, some exclude it, some do both, and most do not say which they do + - currency of amounts. Talk to accountants and come up with a good + way to report this; it is clear that the bulk of sources just want to + use GBP, but some departments that operate overseas have complexities + and no clear exchange rate. + - reporting of VAT numbers is rare. We should reconsider whether this + is worth bothering with + - reporting of dates: invoice date, or payment date? + +Many people included too many or too few fields. Standardise on a set +of mandatory and optional fields. Strongly discourage the inclusion of +data not in the optional set, because people tend to add extra columns +reflecting their opinion of how the data should be structured, and +neglect the recommended ones. 20% of submitted sources used exactly +the recommended columns without prompting; let's get this number up. + +Most people wanted to include a unique transaction reference +number. Add this to the set of standard columns. + +Some people wanted to include a narrative/description field. This +should be encouraged; add as optional field. + +Some people wanted to include commentary or cover notes in their +spreadsheets. This should be strongly discouraged. It needs to be +emphasised that they are supposed to be releasing *raw* data for +analysis, not documents for people to read. + +Invalid data is heavily skewed towards the same errors: + +1. The URL supplied to data.gov.uk does not point to a csv or +spreadsheet. This accounts for about 10% of all entries on data.gov.uk +and in the bulk of those cases, the URL simply points to nothing; the +largest remaining case is a URL pointing to a web page that talks +about the data or lists places it can be downloaded, instead of +pointing directly to the data files. + +This could be eliminated entirely by fetching URLs when they are +submitted to data.gov.uk, and rejecting anything that is not a csv or +spreadsheet. A simple check of the first few bytes of each file is +sufficient to identify almost every error immediately and reject URLs +which are obviously wrong, and this would eliminate over 90% of bad +submissions. No other action could be so valuable in terms of data +gained from time spent, so this should be done first and soon. I would +estimate the engineering cost to be substantially less than a day for +a person familiar with the code. + +A subset of these cases will be URLs that were once valid, but the +files have since been removed. Data sources should be reminded of the +need to maintain a permanent archive of this data at fixed +URLs. data.gov.uk should regularly revalidate URLs and automatically +mail responsible people when they go away. + +2. Automated data extraction/reporting that went wrong - spreadsheets +full of formulas or errors. Automated reporting is a good idea; nobody +looked at these files before uploading them because they are obviously +wrong. It should be straightforward to get them fixed if anybody ever +tells the creator. + +Errors not falling into the above two categories are mostly cases of +complete nonsense or lack of understanding from the data +submitter. These should be handled on a case-by-case basis. + +There is no evidence of widespread difficulty or need for +education. Clear and precise guidelines about the format to release +data in, and validation of submitted URLs, should be the focus. Only a +tiny number of submitters (<10) had an ignorance problem, and these +are likely to be a simple case of the problem being dumped on a junior +employee because nobody thought it was important. + +Areas for further work after everything above this line has been done: + + - unique identification of suppliers. Name isn't very good at + identifying companies, and we should be able to link in other data + about companies + - what value should be in the departmentfamily, expensearea, and expensetype fields? + - character set of submitted data diff --git a/examples/openspending/content/blog/2012-03-12-technology-for-fiscal-transparency-where-next.md b/examples/openspending/content/blog/2012-03-12-technology-for-fiscal-transparency-where-next.md new file mode 100644 index 00000000..eccb2c74 --- /dev/null +++ b/examples/openspending/content/blog/2012-03-12-technology-for-fiscal-transparency-where-next.md @@ -0,0 +1,38 @@ +--- +authors: +- lucy +redirect_from: /2012/03/technology-for-fiscal-transparency-where-next/ +title: Technology for Fiscal Transparency, Where Next? +--- + +## Who is using technology to follow the money? The hunt is on... + +Over the last month, we have been working on a report entitled "Technology for Transparent and Accountable Public Finance" for the Global Initiative on Fiscal Transparency. + +by imtfi on Flickr + +We are hoping to identify the most promising projects around the world that are using technology (web, mobile or otherwise) to further aims of fiscal transparency. Of particular interest are projects that aim to: + + * Publish more or better data related to fiscal processes (aid, revenues, budgets, audits, etc. -- see below), + * Help understand this data through the creation of better visualisation and data analysis tools, + * Educate citizens about fiscal processes, and assist civil society organisations promoting accountable governance, + * Facilitate direct participation in fiscal matters through participatory budgeting, citizen auditing and the like, + * Provide policymakers with complete and reliable data relevant to their work, enabling them to make better decisions. + +We're particularly interested in efforts to improve transparency in 3 main areas: + + * Looking at where the money comes from: In revenue processes (taxation, extractive industry, etc.), + * Monitoring where the money goes: The budgeting process (participatory budgeting, comparisons of planned and retrospective budgets) through to auditing of expenditure, and everything in between. + * The invisible money: projects that aim to improve public understanding of state owned (or semi-owned) enterprises, sovereign wealth funds and contingent liabilities - information on which often are not published as part of current budgeting practices. + +There will be particular focus on the questions 'Who are the users?' and examining their motivations for getting involved, the scalability and applicability of given solutions to other contexts. + +The report will also aim to highlight gaps - so please feel free to think outside the box; if there is cutting edge technology being used in other fields besides public finance, please feel free to suggest it - maybe no-one apart from you has thought of it yet! + +## Over to you + +We are now opening up to the community to let us know if there are any projects we should be aware of and include in the report. + +If you are aware of any projects that we should cover in the report, or if you have any more general observations on the above, please let us know. We have created a [Google form](https://docs.google.com/spreadsheet/viewform?formkey=dGZ1anpCaVZWTTBmR2JQWXFGc0pxeEE6MQ#gid=0) which you can use to give full details and look in more detail into some of the areas we are focussing on. + +For more general comments or observations, and notes of people to contact, please don't hesitate to drop us a line: lucy.chambers [at] okfn.org and velichka.dimitrova [at] okfn.org. \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-03-13-voting-systems-for-e-participatory-budgeting-upcoming-call.md b/examples/openspending/content/blog/2012-03-13-voting-systems-for-e-participatory-budgeting-upcoming-call.md new file mode 100644 index 00000000..96bd236b --- /dev/null +++ b/examples/openspending/content/blog/2012-03-13-voting-systems-for-e-participatory-budgeting-upcoming-call.md @@ -0,0 +1,34 @@ +--- +authors: +- lucy +redirect_from: /2012/03/voting-systems-for-e-participatory-budgeting-upcoming-call/ +title: Voting Systems for E-Participatory Budgeting - Upcoming Call +meta: + _edit_last: "239" +tags: +- Working Group +--- +What are the best systems for voting in tech-solutions to Participatory Budgeting? Join upcoming call to contribute to the discussion. + +The Doodle Poll is out [here](http://www.doodle.com/f6upt8utu6ifds8f#table) - please fill it in if you would like to join to help us schedule the call. + +If you would like to join, please also add your name and Skype ID to the pad [here](http://wdmmg.okfnpad.org/pb-voting) - then I will add you on Skype before the call. As usual, please feel free to add to and modify the agenda. + + + +## Suggested topics for the call + +* Best practice (for different stages of pb process) - current examples +* Preventing: + * Bias in decisions offered + * Undue influence from special interest groups + * Gaming the system + * Herding effects +* Promoting: + * Voting after sufficient deliberation + * Maximum participation + * Maximum information + * Come-back-next time +* Personal profiles and authentication + +I hope you will be able to join us. Please contact me via lucy.chambers [at] okfn.org if you have any questions. diff --git a/examples/openspending/content/blog/2012-05-29-GIFT-report-released.md b/examples/openspending/content/blog/2012-05-29-GIFT-report-released.md new file mode 100644 index 00000000..b3c4b541 --- /dev/null +++ b/examples/openspending/content/blog/2012-05-29-GIFT-report-released.md @@ -0,0 +1,60 @@ +--- +authors: +- lucy +redirect_from: /2012/05/GIFT-report-released/ +title: Technology for Transparent and Accountable Public Finance +--- + +In early March, we embarked on a project to map out projects which use [technology to further the aims of fiscal transparency, accountability and participation](http://openspending.org/blog/2012/03/12/technology-for-fiscal-transparency-where-next.html). Today, we are happy to announce the official release of the resulting report, Technology for Transparent and Accountable Public Finance. Preliminary findings were presented at last month's [GIFT](http://fiscaltransparency.net/) meeting in Brasilia. Since then, we've been building on the comments, follow-up questions and feedback from the session. + +Looking at government revenue, expenditure and off-budget information - we have attempted to identify projects from both governments and civil society which use innovative approaches to: + +* Publish more or better data related to fiscal processes (aid, revenues, budgets, audits, etc. -- see below), +* Help understand this data through the creation of better visualisation and data analysis tools, +* Educate citizens about fiscal processes, and assist civil society organisations promoting accountable governance, +* Facilitate direct participation in fiscal matters through participatory budgeting, citizen auditing and the like, +* Provide policymakers with complete and reliable data relevant to their work, enabling them to make better decisions. + +We focussed in particular on the question: 'Who are the users?'. We examined their motivations for getting involved, the scalability and applicability of given solutions to other contexts. The report also aims to highlight gaps that prevent users from taking up these tools. + +### Report now available online + +Today, the first edition of the report is published on [OpenSpending.org](http://openspending.org/resources/gift/index.html). It is also available for [download as a PDF](http://content.openspending.org/resources/gift/pdf/ttapf_report_20120530.pdf). Accompanying the report is a [project database - bit.ly/TTAPF-projects ](https://bit.ly/TTAPF-projects) which contains many more projects that publish, analyse and demystify fiscal data. + +The section on participatory budgeting deserves special mention. We discovered so many projects that they merited their own listing, which can be found [here](https://docs.google.com/spreadsheet/ccc?key=0AvoV_cBqwo28dE9fZy02NEt2UGxPTnRQMTEzaUhTOGc#gid=4). As we go through, we are building up a catalog of government finance portals in [the 'finance' group of datacatalogs.org](http://datacatalogs.org/group/finance). There's still a lot of work to be done there, but the group already contains the portals mentioned in the report. + +As our work continues, we'd love to maintain these connections and hear updates from the projects and learn about new projects. If you have come across an interesting project and think we should feature it, [please let us know](mailto:gift-report@okfn.org)! + +### Key Findings + +We have tried to highlight specific roles which GIFT could play in promoting the good practice requirements of the report. The slides from the session can be found below: + +
Gift presentation

+ +Read about the highlights in context in the [Highlights, Gaps and Recommendations section](http://openspending.org/resources/gift/chapter1-3.html) + +### Read the report + +See below for a quick overview of the contents: + +* [Chapter 1 - Introduction and Methodology](http://openspending.org/resources/gift/chapter1.html) +* [Chapter 2 - Publishing Fiscal Data: Government Perspectives](http://openspending.org/resources/gift/chapter2-intro.html) +* [Chapter 3 - Using Fiscal Data: Civil Society Perspectives](http://openspending.org/resources/gift/chapter3-intro.html) +* [Chapter 4 - Standards for Fiscal Data: Towards an international framework](http://openspending.org/resources/gift/chapter4-intro.html) +* [Chapter 5 - Case Studies - Where Does the Money Come From?](http://openspending.org/resources/gift/chapter5-intro.html) +* [Chapter 6 - Case Studies - Where Does the Money Go?](http://openspending.org/resources/gift/chapter6-intro.html) +* [Chapter 7 - Case Studies - The Invisible Money](http://openspending.org/resources/gift/chapter7-intro.html) +* [Chapter 8 - Putting the Parts Together, OpenSpending and Publish What You Fund](http://openspending.org/resources/gift/chapter8-intro.html) +* [Final Observations and Review](http://openspending.org/resources/gift/chapter9-intro.html) +* [Further Resources](http://openspending.org/resources/gift/bibliography.html) +* [Appendix](http://openspending.org/resources/gift/chapter10-intro.html) + +### Get involved in the next edition + +This release is version one, and we hope that the research will be ongoing as the OpenSpending community grows and the tools and network develop. As this happens, we'd really love your input. Some suggestions: + +1. Feedback - let us know what you thought of the report and suggest improvements, particularly feedback for GIFT, what role would you like to see them play in this important field? +2. Keep your eyes peeled for interesting projects. We're hoping to feature information about new projects in the blog, so drop a line to the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending) if you know of any we should feature. +3. Help us build up the [finance group on datacatalogs.org](http://datacatalogs.org/group/finance) and review the sites for their usefulness. Ever tried to get fiscal information out of a portal? Did you get what you were after? And importantly, could you use it once you had it? Let us know [here](https://docs.google.com/spreadsheet/viewform?formkey=dGNXNVFXdDlPNlRDaXB2bXc0aGR5UVE6MQ#gid=0). + +Follow up posts on the findings in detail coming soon! \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-06-05-IATI-on-OpenSpending.md b/examples/openspending/content/blog/2012-06-05-IATI-on-OpenSpending.md new file mode 100644 index 00000000..41ec227e --- /dev/null +++ b/examples/openspending/content/blog/2012-06-05-IATI-on-OpenSpending.md @@ -0,0 +1,87 @@ +--- +authors: +- mark +redirect_from: /2012/06/IATI-on-OpenSpending/ +title: Aid Data - From XML to Visualisations +--- + +Are the World Bank and Department for International Development (DfID) spending money on projects in similar sectors and countries? Does all aid to Kenya go the North-East? How much aid in total did India receive last year? + +Until recently, it was impossible to know. But now, thanks to the International Aid Transparency Initiative (IATI), we've been able to start to answer these questions - making the aid process more transparent, which is crucial for making it more effective. + +[IATI](http://aidtransparency.net) is a political agreement by the world's major donors - including international banks, private foundations and NGOs - on a common way to publish aid information. It also defines a technical standard for exactly how that information should be published, IATI-XML. + +So far, 29 donors representing 74% of Official Development Finance (ODF) [have committed to publishing](http://aidtransparency.net/implementation) to IATI. A further [13 donors](http://iatiregistry.org/group) representing 45% of ODF have already published, and 12 NGOs and foundations have published their own data. + +This post details how we converted each donor's data, using simple scripts and open source tools, from raw XML data in the [IATI Registry](http://iatiregistry.org/) into a consolidated dataset and then, via loading into [OpenSpending](http://openspending.org/) to visualisations like those shown above and an easy-to-use RESTful API. + +#### From this.... + + + + +#### ... to this. + + + +## Getting the Data Together + +Full details of how we got the data together are in this case study on OpenSpending ... but to summarize: + +* We grabbed a list of all the IATI data files via the IATI Registry API (the IATI registry is running [CKAN](http://ckan.org/) so this is very easy) +* We converted the data to an SQLite database and a simplified CSV format and posted these on the [IATI dataset on the DataHub](http://datahub.io/dataset/iati-registry) +* Modelled and loaded it into OpenSpending, creating views to visualize it in basic forms. + +## What you can see + +You can now explore the complete dataset of [aid data released so far through IATI, exploring the aggregate and detailed data on OpenSpending](http://openspending.org/iati/). You can drill down through the data and look at it from different perspectives, from exploring the largest sectors in a country, to different implementing organisations in that sector, to looking at all the projects implemented by a single organisation. + +#### Drill down from one layer... + +![IATI 1](http://farm8.staticflickr.com/7092/7341296378_c6ae9b8d6e_z.jpg) + +#### ... to the next - we're zooming in on China here, breaking down by flow type... + +![IATI China Zoom](http://farm9.staticflickr.com/8006/7341296584_1dfbd5ac5a_z.jpg) + +#### ... and you can switch between breakdowns - slicing data here up by organisations implementing the aid... + +![IATI China Implementing Organisation](http://farm8.staticflickr.com/7232/7341296452_857af887ba_z.jpg) + +#### ... and here by funding organisation + +![IATI China Funding Organisation](http://farm9.staticflickr.com/8024/7156094599_a2e8c531e2_z.jpg) + +## More details + +We've also just put together a [briefing on how we worked with the IATI data on OpenSpending.org](http://openspending.org/resources/iati/index.html). The briefing covers in depth what IATI is, using the IATI registry, consolidating data into a simple format, loading data into OpenSpending and using the API. + +## Next steps & get involved. + +For those keen to put coding knowledge to good use to further the IATI mission, some ideas below: + +* Use the API - you can use OpenSpending's API to build applications - read the [briefing](http://openspending.org/resources/iati/index.html) for more ideas and instructions +* [Review our scripts](https://github.com/okfn/iatitools) for converting IATI data. We've been compiling a list of known [issues](https://github.com/okfn/iatitools/issues) with possible future extensions such as geo-coding, reconciling organisations and handling currencies. + +## What's in the data, what's still to come + +The dataset contains current and future spending by major aid donors representing 44% of ODF, with disbursement data running up to the current month in some cases. It also contains commitment data up to 2016 from one donor (and from multiple donors up to 2014). + +However, the data does not contain any information from donors who have not yet published to IATI, and it also does not yet include results, project documents or geo-coded data. + +Future projects might include: + + * Validation - to ensure that data is properly formatted and uses standard codelists; + * Adding results, [geo-coding](http://open.aiddata.org/content/index/geocoding) and project documents to the OpenSpending visualisation - some of this is already available in the original source data, but has not yet been incorporated to this dataset; + * Other visualisations - for example, a map, and activity and transaction views; + * Running the dataset compilation automatically - so that it runs on a server nightly, is up-to-date and imports the latest version to OpenSpending as it's updated. + +## The future + +Eventually what we'd like to see is something like this: an integrated dataset of aid and budgets in each country, so that the full picture of resource flows is available. + +![PWYF Uganda](http://farm8.staticflickr.com/7089/7242685452_5a849c773b_z.jpg) + +**Which country will be next to join up their aid and budgetary flows?** You can get in touch with us via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending) if you have any questions about this project or the data. + +This post was written by [Mark Brough](http://okfn.org/members/markbrough). \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-06-11-opendata-ch-workshop.md b/examples/openspending/content/blog/2012-06-11-opendata-ch-workshop.md new file mode 100644 index 00000000..90dffd8d --- /dev/null +++ b/examples/openspending/content/blog/2012-06-11-opendata-ch-workshop.md @@ -0,0 +1,14 @@ +--- +authors: +- matthias +redirect_from: /2012/06/opendata-ch-workshop/ +title: Workshop - Open Budget and Procurement Zurich June 28th 2012 +--- + +**As part of the Opendata.ch conference on June 28th 2012 in Zürich there will be a workshop dedicated to the topic of open budget and procurement.** + +Various speakers from Switzerland and Germany will make short presentations and the discuss with the audience the implications and possible future actions. First, Friedrich Lindenberg of the Open Knowledge Foundation will share an overview of OpenSpending.org and present case studies of open budget initiatives. Second, Maja Menn, head of the finance department of the city of Zürich, will share her critical thoughts on open government data for public finances. + +Then three speakers, Christian Geiger of Zeppelin University, Andreas Burth of University of Hamburg, and Alexandra Collm of University of St. Gallen, will provide insight into their scientific research on open budget. Last but not least two software developers, Thomas Preusse and Daniel Meister, will show examples of new open budget applications: city budget of Bern and public procurement data of the Swiss platform [simap.ch](https://www.simap.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=1&page=/MULTILANGUAGE/simap/content/start.jsp&language=EN). + +To participate in this session please sign-up for the Opendata.ch conference [here](http://opendata.ch/2012) Please note: The session will be held in German. diff --git a/examples/openspending/content/blog/2012-06-11-upload-videos.md b/examples/openspending/content/blog/2012-06-11-upload-videos.md new file mode 100644 index 00000000..a777c2d6 --- /dev/null +++ b/examples/openspending/content/blog/2012-06-11-upload-videos.md @@ -0,0 +1,42 @@ +--- +authors: +- lucy +redirect_from: /2012/06/upload-videos/ +title: Video Instruction Guide - Loading Data Into OpenSpending +--- + +Recently, the OpenSpending team have been working on a project to visualise financial data in Cameroon. One of the aims of the project is to create a platform which is sustainable for years to come and that means that it needs to be really easy to load and maintain datasets into OpenSpending. + +So... we've created some screencasts about how to load data into OpenSpending. Please do take a look and try it out for yourself and let us know if anything needs to be clearer! + +Once you've got your data into the [OpenSpending data format](http://openspending.org/help/data-cleansing.html) you're ready to load. + + +## Preparing your dataset information + +First add information about your dataset to make it easily findable in OpenSpending + +

OpenSpending Upload from Open Knowledge Foundation on Vimeo.

+ +## Upload your data online + +Next, you need to get your CSV file online. There are numerous ways to do this, [Dropbox](https://www.dropbox.com/), exporting your data directly from a Google Docs (*File > Publish to the Web > Publish a CSV*). We've used the DataHub to publish our data. If you've never used it before here's a quick demo of how to upload a dataset. + +

DataHub upload from Open Knowledge Foundation on Vimeo.

+ +## Create a model to explain the structure of your data to OpenSpending + +Next, tell OpenSpending how to understand your data by creating a model. + +

OpenSpending - Create a Model from Open Knowledge Foundation on Vimeo.

+ +If that's all gone well, you'll be able to move on to creating visualisations (video coming soon - watch this space!) and if not and you have some errors watch the video below to see what to do next. + +## Not quite right? How to fix errors... + +It doesn't always go right the first time you try and load your data - here's what to do if you have made a mistake either in your model or your data. + +

OpenSpending Error from Open Knowledge Foundation on Vimeo.

+ +Please let us know if you upload a new dataset and if you have any feedback. You can get in touch any time via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). + diff --git a/examples/openspending/content/blog/2012-06-28-OKFest-announce.mdown b/examples/openspending/content/blog/2012-06-28-OKFest-announce.mdown new file mode 100644 index 00000000..e62c0dd3 --- /dev/null +++ b/examples/openspending/content/blog/2012-06-28-OKFest-announce.mdown @@ -0,0 +1,33 @@ +--- +authors: +- lucy +redirect_from: /2012/06/OKFest-announce/ +title: Budget Cycle Monitoring Hackday at OKFest +--- + +**Often, the only way to check whether governments are releasing financial documents, is retroactively. But what if it were possible to receive alerts and check up against internationally recognised best-practice guidelines as to when the government in your country should be releasing key budget documents?** + +At present, the only way to find out when to expect a document to be released in a given country is to trawl the legislation of a country and find references to legally mandated release dates, but there may be another way... + +At this year's [Open Knowledge Festival](http://okfestival.org/) in Helsinki, there will be a 'Budget Cycle Monitoring Hackday' - to build a prototype of a calendar to attempt to fix exactly this type of problem. + + + +## Who is it for? + +We figure journalists and researchers need fiscal data as soon as it is released. NGO's monitoring budget practices need to have a way to check when budget documents are released. We hope these groups and more would benefit from the calendar and notification services. + +## About the Hackday + +**Mission:** To align a calendar of internationally-recognised best practice guidelines for publication of key budget documents with calendars of the fiscal year in different countries and build services which can relate to this. Possible features include: + +1. The ability to send out notifications to journalists, CSOs, budget monitoring orgs +at the last acceptable date for the docs to be published to check whether they are available. +2. Integrating with FOI request services to allow organisations to have a mechanism to request budget documents directly. For example, auto-generated FOI letters processed through services such as [Alaveteli](http://www.alaveteli.org/). + +## How to join + +Simply [sign up for a ticket for OKFestival](http://okfestival.org/tickets-and-bursaries/), bring a computer and join us on the day. + +Discussions will be ongoing about what features to include, via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending) - if you know someone who would benefit from this project, we'd love to hear from you! + diff --git a/examples/openspending/content/blog/2012-07-02-tester.mdown b/examples/openspending/content/blog/2012-07-02-tester.mdown new file mode 100644 index 00000000..851754d9 --- /dev/null +++ b/examples/openspending/content/blog/2012-07-02-tester.mdown @@ -0,0 +1,9 @@ +--- +author: $authornamehere +redirect_from: /2012/07/tester/ +title: Test +--- + +Tester +======= +Tester diff --git a/examples/openspending/content/blog/2012-07-05-OSI.md b/examples/openspending/content/blog/2012-07-05-OSI.md new file mode 100644 index 00000000..0c610ef9 --- /dev/null +++ b/examples/openspending/content/blog/2012-07-05-OSI.md @@ -0,0 +1,44 @@ +--- +authors: +- lucy +redirect_from: /2012/07/OSI/ +title: Athens to Berlin - a European Financial Profile +--- + +**How does the work of budget monitoring organisations, investigative journalists, academic researchers and think-tanks differ in the various countries of Europe? What are the key questions these organisations address - are they largely the same in every country, or does each country have a different issue at the forefront of their minds? What tools and techniques do they have at their disposal to get to the bottom of Europe's tricky financial situations?** + + + +A few months back, [we wrote about our project to map out how Civil Society Organisations around the world use technology in their work](http://openspending.org/blog/2012/01/12/civil-society-and-spending-data-who-is-mapping-the-money.html). Having identified in the first phase numerous interesting organisations working in this area, we now move on to phase two, getting down to the questions of tools, methodologies and barriers. A couple of the questions we will be looking at can be found below: + +* How government financial information relates to the mission of CSOs, what questions are they trying to answer? +* What are the high-value datasets? Besides the eight key budget documents that every government should produce, as highlighted by the International Budget Partnership, which datasets should we be asking that governments of every country in the world to release? +* How do CSOs get hold of the data? Formats, channels & procedures. +* How do they ensure the sustainability of their efforts? What happens if the person who has been working with the data leaves? Do they document their processes? +* What tools are used to work with, archive, share and to spread the word about any findings? +* Anywhere else the conversation logically takes us... + +Considering Europe's current financial situation and the Euro crisis, my focus for this current section is a cross-section of Europe - from Greece to Germany. These two countries stand at either end of the debate on proposed austerity measures for the Euro Zone, but are they actually poles apart in terms of challenges they face in monitoring their budgeting priorities at a grass-roots level? or are there common issues which could be solved with a technical helping hand? + +So, armed with a video camera, sunscreen and an InterRail ticket, I am departing today to find some of the answers to these questions and will visit Greece, Romania, Hungary, Slovakia and (possibly) Czech Republic collecting opinions on what it is like to be working with government financial data across Europe... + +This chain will be completed in the autumn with a trip to Germany, to whom most of Europe is currently looking for hopes of salvation of the common currency. + +## First Stop - Athens + +The first thing that I should note about my journey to Greece is that, unlike the other countries I intend to visit, I have been, as yet, unable to find a Civil Society Organisation which specifically dedicates its effort to monitoring government budgets, contracts and procurement. Any hints welcomed if such an organisation exists, please do put them in touch!ß + +However, I will meet the team behind **Diavgeia** - the Government "Cl@rity" programme, to talk about the up and coming project **"AGORA"**, which deals with transparency in the cost of supplies to the Greek Public Administration Nikolaos Stavropoulos, Maria Galaktopoulou (the Vice Mayor) and Photios Zygoulis. Secondly, Thodoris Papadopoulos, a member of the OpenSpending community and graduate of the **National School Of Public Administration**, who has run up against problems in using Greek data has promised to share his thoughts on what could be done better in Greece. Finally I will meet the team of **Transparency International - Greece**, in particular, Ms Effie Vraniali, a lawyer and a PhD holder in Public Financial Management, before moving on to Thessaloniki. + +Our aims in this phase are threefold: + +1. To work out what tips and tricks the various CSOs could learn from one another, to document them and put them together as a Spending Data Handbook. +2. To work out what tools are being used in budget and spending monitoring - are there any gaps which could easily be filled? +3. To connect the civil society organisations together, to ensure that ideas keep being exchanged. + +## Stay in touch + +Please do get in touch and let me know of any questions you would like me to put to the CSOs. The easiest way to stay in contact is via the [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). + +*Image credits: [quapan on Flickr](http://www.flickr.com/photos/hinkelstone/)* + diff --git a/examples/openspending/content/blog/2012-07-11-Greece.mdown b/examples/openspending/content/blog/2012-07-11-Greece.mdown new file mode 100644 index 00000000..86e83dbd --- /dev/null +++ b/examples/openspending/content/blog/2012-07-11-Greece.mdown @@ -0,0 +1,55 @@ +--- +authors: +- lucy +redirect_from: /2012/07/Greece/ +title: Athens to Berlin - Day 1, Diavgeia +--- + +**This is the first update from the project: *[Athens to Berlin - a European Financial Profile](http://openspending.org/blog/2012/07/05/OSI.html)*** + +In the sweltering heat of downtown Athens, I visit the heart of Greece's new transparency initiative - Diavgeia... + +# Diavgeia - Transparency, for Governments' Sake + +Greece has historically been known for its lack of financial transparency. The [2010 report](http://internationalbudget.org/wp-content/uploads/2011/06/2010_Full_Report-English.pdf) by the International Budget Partnership cites an article from the OECD Journal on Budgeting: + +> “[Greece's] reported budget balance was affected by off-budget military spending and +overestimated surpluses in social security funds.” + +and then a BBC article from 2004: + +> "Such flaws in its budget reports actually enabled Greece to join the euro currency in 2001 because it misreported its fiscal deficit numbers, claiming a budget deficit in 1999 that was less than three percent (a condition required to be met by countries wishing to join the common currency) when in fact its budget deficit exceeded that target by a substantial margin" + +The International Budget Partnership decided not to include Greece in the 2010 Open Budget Survey, citing: + +> "[Some problems] such as the weak legislative oversight, would likely have been illustrated by the Survey. Many of the problems, however, relate to the inaccuracy of the information reported and the ongoing need for subsequent revisions, which would not have been directly captured by the Survey." + +With the current state of the Eurozone, it is clear that Greece needed to take control of its public financial management practices and its proposed solution is a topic of much conversation and the first stop on my journey... + +# Everyone's talking about: Diavgeia (Cl@rity) + +[Diavgeia](http://diavgeia.gov.gr/) (English: "Cl@rity") is the Greek Government's program to cut down on paper records and digitise documents related to the processes of government. Since 1st October 2010, all ministries have been obliged to upload, according to the claim on the website, **every government decision** to Diavgeia. This includes information on companies they contract with down to, as my first interviewee, Nikolaos Stavropoulos, claims "information on expenditure however small, even a plastic glass or pencil purchased should be counted". Diavgeia also serves as a platform for online deliberation, every draft legislation or policy initiative, it is also the centralised place for advertisements of openings in public office. + +## E-Government in Neo Iraklio + +Nikolaos is Scientific Advisor in the office of e-Government - [Municipality of Neo Iraklio Attikis](http://www.iraklio.gr/), Athens, which I am led to believe is the first municipality to build municipal level services on top of Diavgeia. The services include 'Fix My Street' style applications, where citizens can feed back on the services provided by governments as they see them, and also forms a place to discover what services are offered in the local area. Even though the application is only about 3 months old, it has been well marketed at the local metro station and already receives around 35 requests per day. The application recently won an award at the European Data Forum in Denmark. When I get on a better internet connection - I'll upload the video of the presentation Nikolaos gave me. + +While I'm in the office, I'm also given a tour of the user interface for officials, via which the data is 'born' - more comments on this in my next post. + +The efforts of the municipality clearly provide a useful resource, both inside government, to help to know where do focus efforts, resources and staff, as well as outside, where citizens are given a clear channel to contact the government and a lot of information on who is responsible for a given decision. Too early for comment yet, but we wait to see how this project will develop. + + + +## Diavgeia - more than a platform + +Back to Diavgeia at the national level. Diavgeia is more than a platform, it is an entire workflow. Decisions are not effective in law unless they are uploaded to the platform and public officials (and their superiors in chain of command) are held personally accountable for entering the information, which should be done in as near to real time as possible - ideally within 24 hours. The project is still in its infancy, and Nikolaos explains the need not only for education in the civil service for how to use the platform, but also a change in mentality - most civil servants are largely used to dealing with decisions on paper and this new level of participation and transparency will take a bit of getting used to. + +I also hear rumours (not in the e-Gov office) that Greece does not know know exactly how many civil servants it has, but estimates circulate around 1/10 of the population working for the public service. The benefits of having information in a digital format are clearly being recognised, and Greece is expending its efforts to reap the benefits of digital information. + +Diavgeia is a good example of how transparency is not just about civil society holding government to account. Transparency is equally important to civil servants, which often suffer equally from lag times in getting information, and having a centralised resource is often a good way to ensure *everyone* in government has access... + +My next stop was to find people who actually used Diavgeia to get their feedback: the team behind [publicspending.gr](http://publicspending.gr/) and a lawyer working with Transparency International, Greece. As I mentioned in my announce blog post, I've had difficulty in finding conventional budget monitoring organisations. + +My next blog post, the provisionally entitled 'The Chicken and The Egg: Where are the budget monitoring organisations in Greece?' focusses on these users of Diavgeia to get their perspective on how it is useful and whether it can be improved... I'll be back soon. + +*I am also working on a full profile of Diavgeia, similar to those produced for other countries as part of the [GIFT report](http://openspending.org/resources/gift/chapter2-intro.html), keep your eyes peeled on [OpenSpending.org](http://openspending.org/).* diff --git a/examples/openspending/content/blog/2012-07-16-Greece-2.md b/examples/openspending/content/blog/2012-07-16-Greece-2.md new file mode 100644 index 00000000..f506a5f8 --- /dev/null +++ b/examples/openspending/content/blog/2012-07-16-Greece-2.md @@ -0,0 +1,68 @@ +--- +authors: +- lucy +redirect_from: /2012/07/Greece-2/ +title: Athens to Berlin - Chicken or the Egg? Days 2-4 (Athens and Thessaloniki) +--- + +Immediately after landing in Greece, I have my first experience of where austerity cuts were hitting services. While attempting to buy tickets for the metro, I struck up conversation with an irate but lovely lady named Tina, who was irritated to have to travel all the way into central Athens by metro just to travel out again on another line to get home, a journey she would normally do via the suburban railway, which wasn't running, meaning the journey would take twice as long. *"Cuts!"* she told me in a mock-angry voice. + +One of the things I found quite bizarre in Greece is that many people seemed to be able to put a date on a lot of the public works which had taken place: + +> "Sorry about the pavements here, the last time they renovated them was 2004." + +2004, came up a lot and it was only later that I realised that that was the last time that Greece hosted the Olympics; I suspect in 10 years I will be saying similar things about London... + +All of this gets me thinking about whether transparency has anything to offer in terms of mitigating the effects of the cuts, if people could see why and where the money was disappearing from services, would it help to make the unpopular decisions faced by policy makers less difficult? + +## Chicken or the Egg - where are the budget advocacy groups? + +As I mentioned in my announce post, I've not come across *any* budget advocacy groups in Greece. I ask everyone I meet, but no-one can point me to any... I eventually come across [Greece Debt Free](http://www.greecedebtfree.org/), a crowdfunding project to volunteer away Greece's debt, but by that point I am already leaving, so I make a note to get in contact ... I start to wonder whether this is because before Diavgeia, information on public spending simply was not available and so the work was not possible, but was there genuinely no-one campaigning for the information to be released? What comes first? Budget advocacy groups or budgets? + +On the off-chance that they knew anyone working in this area, I dropped a line to the local Transparency International group. While they also did not focus on the issue, they put me in touch with a lone-ranger, one of their lawyers, Effie Vraniali, who studied the Greek Public Financial Management System as part of her PhD, comparing it to other countries. Below are some of her thoughts. + +Highlights: + +* Not only absence of the data, but finding out who was responsible for what within government was an issue, +* Even the Greek Parliament themselves struggled to get hold of the information they required, +* Information in Diavgeia was quite scattered and thus unusable unless you had the time to search for everything you wanted + +

Effie Vraniali - How Greece manages its money from Open Knowledge Foundation on Vimeo.

+ +Last but not least, Effie mentions that she would *love* to do more work to measure the openness financial information in Greece, but she can't do it alone, she needs a team. Effie is now part of the [Working Group on Open Spending Data](http://openspending.org/resources/wg/index.html), so if you are likewise looking to work in this area in Greece, please [get in touch via the mailing list](http://lists.okfn.org/mailman/listinfo/openspending). + +## Diavgeia's Dodgy Data? + +Everyone who I speak to is full of praise for what the Greek government are doing with Diavgeia, although all have comments on how it could be done better! + +While I'm still in Athens, I meet up with [Thodoris Papadopoulos](http://thodoris.net/weblog/archive/2012/02/29/?p=176), who wrote a paper at the National School of Public Administration on the pros and cons of the Diavgeia system, which he promises to translate selections of and send me. I also meet [Ioannis Anagnostopoulos](http://www.anagnostopoulos.name/) from the Athens team of the cross-university collaboration team [PublicSpending.gr](http://publicspending.medialab.ntua.gr/) who shares his experiences on how easy it was to use Diavgeia to build their research platform. + +In my next post, I'll urge journalists, researchers etc to have a look at Diavgeia's data, but before I do, here's a couple of things they flagged up to be wary of... + +### How is data born? + +The manner in which data is generated affects how it can be used, here are the thoughts I collected from Thodoris, Ioannis and others... + +* **Data is uploaded in various formats, often PDFs**. When I was in the e-governance office of Neo Iraklio Attikis, I was given a demonstration of how the data was uploaded to Diavgeia, it took quite a while to locate and upload individual PDFs to the form they were supposed to support. In the back of my mind, I keep thinking to myself, "Why do people like rubber stamps so much?", surely there is a quicker way than this? As has already been pointed out, this also makes it very difficult to search across all of the data in the site at once. +* **A couple of the most important fields in the civil servants' form used to be free text, and not mandatory.** e.g. Some people may have used "." instead of commas, some may use the word "Euro" rather than the sign "€". It was fixed after repeated notices from developers. (Go civic-minded-developers!) +* **Information in the *amount* field is stored as text, rather than as numbers.** A deeper complication of the problem above. + +This is particularly problematic when combined with another procedural problem. There are 3 stages to making any official purchase: + +1. A document has to be signed to show the expense has been accepted (like a purchase order) +2. There has to be a decision to actually pay the money e.g. upon satisfactory completion of a task +3. There is the actual payment or transaction + +According to Thodoris, this field is often entered incorrectly and is not mandatory, so it is often impossible to know whether the information you are looking at is a purchase order or the actual transaction. This could cause big problems down the line, not just for CSO budget monitors, but also within government... + +* **The system does not use international standards such as COFOG to classify the information.** This seems to be a recurring theme. For example, if you are interested in a question such as 'how much does the government spend on mobile phones?' it is impossible to find out.Someone buying a mobile phone for the ministry of finance will register it with one code - for another ministry, it will be recorded against another code. +* **A lot of errors.** Diavgeia does not have a validation service. PublicSpending.gr seeks to track down these errors and correct them by validating them against another service, [taxisnet](http://www.gsis.gr/). Ioannis says that 70% of the effort behind PublicSpending.gr is taken up with data cleansing, rather than producing visualisations etc. +* **Missing information.** All of my interviewees were skeptical that *all* information which is meant to actually made it into Diavgeia. Some even had personal anecdotes to prove it didn't. The following quote is from a scientist at the local university, George: + +> "My own experience of Diavgeia. I was giving some lectures in a university for a small amount of money, €60 /hour or something like that. + +>After 2 months they called me back to sign again the contract to say that I was happy for them to publish this information in Diavgeia. After 2 months I discovered through a variety of things that the information had not been added to Diavgeia." + +## On to Thessaloniki + +It may take a group of techies to do it, but all is not lost... my next stop is to meet the WebScience Master Programme at the University of Thessaloniki [Read the profile of the programme](http://openspending.org/blog/2012/07/16/Greece-3.html). diff --git a/examples/openspending/content/blog/2012-07-16-Greece-3.md b/examples/openspending/content/blog/2012-07-16-Greece-3.md new file mode 100644 index 00000000..707677bd --- /dev/null +++ b/examples/openspending/content/blog/2012-07-16-Greece-3.md @@ -0,0 +1,42 @@ +--- +authors: +- lucy +redirect_from: /2012/07/Greece-3/ +title: Athens to Berlin - PROFILE - Linked Data and Public Spending +--- + +From the hot, dusty streets of Athens, I trundle in to breezy but baking Thessaloniki on a late-night train... + + + +In the apparent absence of any budget-monitoring or advocacy groups in Greece, a group of programmers are working directly with the Diavgeia data to present it in interesting ways to make it easy for anyone to understand, rehashing the data with data from new sources (criminality rates, population numbers) and archiving it. + +I'm here to meet the team from Aristotle University of Thessaloniki WebScience Master Programme, which aims to monitor in real time all Greek public expenditure. + +The team have taken a [Linked Data](http://en.wikipedia.org/wiki/Linked_data) approach to combining datasets. New to linked data? Fear not, Professor Ioannis Antoniou from the WebScience Master Program explains, with the aid of some Greek Philosophy what it is: + +

Ioannis Antoniou explains Linked Data from Open Knowledge Foundation on Vimeo.

+ +The whole effort around visualising and working with the data here revolves around Linked Data. The team extract structured information from Wikipedia using [DBPedia](http://dbpedia.org/About) and mix it up with information from [Diavgeia](http://diavgeia.gov.gr/) to produce their new graphs. + +## Meet the team + +For a quick overview of who is who in the team and what they have been working on, I've put together this rough introductory video: + +

Linked Data and Public Spending at Aristotle University, Thessaloniki from Open Knowledge Foundation on Vimeo. + +### Use the data + +The team behind the project is small, but they are interested to hear from anyone interested in using the data. What questions would you like to ask? What datasets could be combined next? + +You can get in touch via the mail above, or the [OKFN Greece mailing list](http://lists.okfn.org/mailman/listinfo/okfn-gr). + +The team are also planning a hackday on budget visualisations in the beginning of October! There will be more details on this soon via the [OKFN Greece blog](http://gr.okfn.org/blog/). + +## More visualisations + +For more visualisations of Greek finances see: + +* [GreekSpending](http://greekspending.com/). +* [PublicSpending.gr](http://publicspending.medialab.ntua.gr/) - an initiative conceptualized, analyzed, implemented, hosted and operated at the Multimedia Technology Laboratory (School of Electrical and Computer Engineering, National Technical University of Athens). + diff --git a/examples/openspending/content/blog/2012-07-19-Caring-for-your-neighbourhood.mdown b/examples/openspending/content/blog/2012-07-19-Caring-for-your-neighbourhood.mdown new file mode 100644 index 00000000..cb91bf81 --- /dev/null +++ b/examples/openspending/content/blog/2012-07-19-Caring-for-your-neighbourhood.mdown @@ -0,0 +1,28 @@ +--- +authors: +- gisele +redirect_from: /2012/07/Caring-for-your-neighbourhood/ +title: Caring for My Neighbourhood - Geolocating Spending in Brazil +--- + +**This post was written by Gisele Craveiro, of the University of São Paulo, member of [OKFN Brazil](http://br.okfn.org/) and one of the coordinators of GPoPAI (Research Group of Public Politics in Access to Information).** + +The public budget should express the population's needs and priorities and its implementation should be as transparent as possible. In Brazil, the municipal budget implementation details must be published on the web daily, but even in the case where this law is acted upon, the reality is that very few people understand them. + +The ["Caring for my neighbourhood"](http://www.gpopai.usp.br/cuidando) project wants to provide means for society to know the budget thematics by better spending oversight. + + + +To achieve the objective, all expenditure related to public equipments in São Paulo are geolocated and shown in a web site. This will support training activities in the community. We aim to promote citizen engagement by showing the user which projects can be found in their area. + +By providing an easy visualisation of many individual expenses placed in a map, it may lead people to make a link between governmental action and something tangible of their everyday life. The tool shows on the map: the expense description, the amount of resources allocated to it and the amount spent so far. Thus data will be more understandable and the resident could take control of what is happening in his/her neighbourhood. + +We hope that the comparison to other areas in the city can give to the community/citizens more skilled arguments during the budget formulation and other decision making processes. We hope that it can contribute to better income distribution and a more efficient fight against corruption. + +Besides the tool, we will develop content about public budget concepts in order to support activities in the community. We will also organize mapping fests so participants can know better the neighbourhood and public equipments that are receiving investment.We intend that the collected information (maps, photos, videos, texts), produced during these activities or later, can constitute a crowdsourcing platform for future monitoring and also feed open platforms like OpenStreetMap. + +Researchers from University of São Paulo (also OKFN members) and Our São Paulo Network (a network of over 600 civil society organizations operating in the municipality of São Paulo) are organising this initiative, but we´d like to invite anyone interested to contribute: sending suggestions, coding or just disseminating this idea/project to whom it may concern. More information with Gisele Craveiro (giselesc at usp dot br). + +The tool beta version can be found at: (only in Portuguese) +Code available in + diff --git a/examples/openspending/content/blog/2012-07-20-Romania.md b/examples/openspending/content/blog/2012-07-20-Romania.md new file mode 100644 index 00000000..784b0aca --- /dev/null +++ b/examples/openspending/content/blog/2012-07-20-Romania.md @@ -0,0 +1,110 @@ +--- +authors: +- lucy +redirect_from: /2012/07/Romania/ +title: Athens to Berlin - Romania's Money Mappers. +--- + +Although the people I meet here in Romania seem to dispute the fact - Bucharest has a very strong NGO scene. Corruption, both institutional and petty, comes high on the focus agenda as well as making citizens vaguely data-literate about how the government spends money and getting a good grasp on where the money comes from. + + + +Andra Bucur from the Soros Foundation has kindly gathered together a group of some of the key players in the world of financial transparency together for a discussion on how . It's clearly a hot-topic and turns into more of a debate than an interview, which I am naturally delighted about :) + +While I'm here, I meet: + +* Andra Bucur - *[Soros Foundation, Romania](http://www.soros.org/about/offices-foundations/soros-foundation-romania)*, working on transparency in Revenues from Extractive Industries +* Bogdan Manolea - *Executive Director, [APTI (Association for Technology and Internet)](http://www.apti.ro/)* +* Cristina Lupu - *[Centre for Independent Journalism](http://www.cji.ro)* +* Codru Vrabie - *[Bribe Market](http://www.piatadespaga.ro/)* +* Elena Calistru - *[Lost Money (Bani Pierduti)](http://banipierduti.cloudapp.net/)* + +Unfortunately, they couldn't join us on the day, but I also make a note with myself to check in with [Expert Forum](http://www.expertforum.ro/) and [Active Watch](http://www.activewatch.ro/), who focus on in-depth analysis of financial information. + +## Highlights + +### Lost Money + +Elena Calistru's project, Lost Money, attempts to engage the public with the scary numbers behind the public budget. She says: + +> "The public budget is something which is very complicated. For the average Romanian (and not only) it is usually this black hole where their money is spent, for roads, for public services and so on, without knowing exactly what happens there. So this platform tried to put everything there but in a very user-friendly manner with beautiful visualisations and by offering the user a personal experience of that complicated process of collecting taxes and spending them..." + +I learn all manner of things about how budgets are calculated in Romania, for example, for school spending, standard costs per pupil have been introduced, with coefficients altering the amount based on the temperature of the region... There's also a *big* debate about whether the favourite topic - roads - should have standard prices per kilometre of road, would that actually solve anything, and would the average citizen care what was spent, as long as they could get from a to b? + + + +Finding the right triggers to engage users is definitely a challenge and there's a consensus that Romanian citizens will only be interested if the data is given a personal angle. Elena's project contains a personal [tax calculator](http://banipierduti.cloudapp.net/calculator/), allowing users to input their salary and get an estimate for how much they contribute in taxes to various spending areas. But the bigger picture and research into big investments is still difficult to engage the public on. Here's Elena on one piece of feedback she got from a piece of her research: + +> "We focused on several contracts, large investment contracts, and we showed that the road from Bucharest and the road surrounding Ploieşti, they paid a huge amount of money [...] I received a phonecall from one user... telling me: "Look, stop it with this, I don't care how much money they've spent here, it's the best road available here, I don't care". [...] It's not correct, they could have done more with that money..." + +Participation is also something which comes up a lot as something the group are keen to promote, Elena describes how she tackles it in her project. + +> [...] "We are now working on a feature where they can become a budget heroes [...] they will be able to see there that this law means cutting expenses of 1 bn Euros from here [...] These are the other policy options [...] Even though I cannot promise the users that all these options you are expressing here will be heard at the decision makers, what I can do is expose them to different policy options for a particular decision and tell them, these are the pros and cons for each of them, what's your choice?" + +*Elena is currently working on an English profile of her project, which we'll publish when it's done.* + +### Money, Media and Megalomania + +Christina Lupu, from the Centre for Independent Journalism described the reason that she currently needs access to information about how the government spends its money: to ensure the media does not become reliant on state support. Christina is working on a project to see how money from the public budget is spent on advertising... : + +> "[I]n a lot of the cases the authorities, the local government, uses money from the local budget buying advertising in the media companies. It is a crisis period. The state in Romania is one of the most important investors in the media, buying advertising from public money; they buy advertising and they say, "OK,you are not allowed to write bad things about me because I will cut the budgets", [...] we had cases that it was written that "you are not allowed to write bad things about me". But now it is something that you know. [The agreement is] 'I will not write it in the contract that you have to write good things about me, that you have to come to my events and present them as the most interesting in the world, but if you don't do that, I will go to the competition...' So the media is powerless with regard to how the money is spent." + +### Bribe Market - Combatting Corruption + +Petty corruption, isn't 'fiscal data' in the pure sense. However, if getting access to basic services requires money to be handed over regularly enough, it becomes almost like an unofficial tax. I listened to Codru Vrabie on his project *[Bribe Market](http://www.piatadespaga.ro/)* (not yet launched) -treating petty corruption as such in an attempt to tackle it: + +> "[W]e looked at bribery as in a sense, a legitimate transaction and tried to see, from a supply and demand perspective how can you tackle that, so we thought, "OK, what if we put together a platform which will help people learn what's the actual rate, so open up the prices, and hopefully also get an indication with smiley faces or a star system as to how satisfied they were as to the services they got?", so you open up information about the quality of service as well. [...] So it's wishful thinking based on economic theory but who knows, maybe in a few years or so, maybe we'll actually get to see results either in the sense that prices of bribes will drop significantly [...] or we will see that corrupt public servants will form a kind of cartel and fix prices at a specific rate, but then if people continue to pay that price that means that that is a legitimate tax for getting that specific service, so what we can do then is go to Parliament and advocate that that specific tax is included as a legal price for that particular service. And then no-one will have any incentive to pay a bribe, because they legally pay a tax." + +I asked Codru whether he had looked at other bribe spot sites such as [I Paid A Bribe](http://www.ipaidabribe.com/) or [BribeSpot](http://bribespot.com/) - he explained that to some extent, he had modelled his site on Bribespot, but there was one key philosophical difference - Bribemarket is also about user satisfaction with services, their *value for money*, in a sense. Users will be provided with a feedback form, where they can write about how happy they are with a given product or service. Two birds with one stone - tackling petty corruption and (hopefully) improving service delivery... + +*We will follow up with Codru to get a full profile of the project when it is launched in the autumn...* + +A slight digression, but while we're on the note of corruption, there were some very interesting examples floating around of how Government technology had been successfully used in Romania to counter corruption, for example, in the passport service. According to the group, where before the slow wheels of the passport system may often have been greased to accelerate the lengthy process of getting your documents, the new digitised passport service cuts out many middlemen, and is faster, so it is almost impossible to pay bribes. + +### Getting the data - what's it like in Romania? + +How do you get data in Romania? How are they finding Freedom of Information laws working out in practice? Would it be helpful to have a portal such as "[What Do They Know?](http://www.whatdotheyknow.com/)" or "[Ask the EU?](http://www.asktheeu.org/)" in Romania? + +> *Andra Bucur*: I usually make information requests and send them to the specific authorities but normally I don't get an answer in 10 days, which is the time specified in the law, sometimes you get in about 30 days but they should advise you beforehand if it takes a while to get your request processed... Usually in the extractive industries, we don't get an answer about the information we want [...] so then we use strategic litigation in order to get the information. [...] We send it by fax, I never sent it electronically. There are institutions that don't even have their contacts on the website. Like the National Agency for Mineral Resources, they don't have budgets online, they don't have public acquisitions online, you can click on everything you want but it doesn't work, they don't have contacts online, so the only way to send a request is to actually send it by post or to go to the institution, it depends on the agency. [...] + +> *Cristina Lupu*: You may submit your requests online, but the problem is if you do that, you can't prove that they received the email. And this is one of the excuses that they use. If you go to the registration office and they put the date and the stamp you are sure that they received the information [...] + +This obviously gets a raised eyebrow from me - rubber stamp syndrome strikes again... We don't come to a consensus on whether the portal would be useful in the end, but we've had a jolly good debate in the meantime... + +The age-old ugly beast of data formats (PDFs) also rears its head again, but not just PDFs - nightmare PDFs: + +> *Codru Vrabie*: The ones I like [this is presumably sarcastic] best are the PDFs that is locked that was actually distilled from a JPG that is actually a copy, tilted like that [gestures], of a third hand fax, so all the letters are smudged and effaced... + +Someday soon, I hope this problem will no longer be an issue, but there needs to be a serious mentality change in government and there need to be more CSOs, such as these, keeping the pressure on to ensure the message gets through... + +#### So what data do they want? + +The group have actually put together a list already of key datasets, when they were consulted about how Romania should approach the Open Government Partnership, they promise to translate and send it on to me. Until the full list is ready, here's a quick overview of some of the things they ask for: + +> *Codru Vrabie*: **Public Institutions - Activity reports.** "For instance the General Authorities Office, the public prosecution. When they published their report a couple of years ago they said there's [X] million criminal cases being tried every year. This is a piece of information which you will find in the annual activity report, it should be referenced to a database. And so, what I'm saying is, for all the information you put in your activity report, give me the attachment, so to say...in an open format." + +> *Elena Calistru*: **Machine-readable budgets - "both at central level and at the local level"** - There are some core datasets which need to be there, which are already in place at least at a central level. [...] at the level of Ministry of Finance, sure you have the budget in a machine readable format, but you're not putting it on your website in a website in an open format, which is silly, you could do this without any effort, you can do it without any costs... and so on, so this is very important to start because it's very, very easy. It's the same with the datasets from the National Institute of Statistics." + +

TOP TIP: Looking for statistics on Romania? You may well be able to find many of the datasets which the Romanian National Institute of Statistics charges for, for free on Eurostat.
+ +> *Elena Calistru*: "**Statistical indicators which are of use for the business sector**, deciding where to invest, what to focus on and so on..." + +> *Codru Vrabie*: **Public Procurement Data** "This is a question you should take to Slovakia - I think they have now 3 different portals from [Fair-Play Alliance](http://www.fair-play.sk/index.php) and [Transparency International in Slovakia](http://www.transparency.sk/) that relate to the public contracts and the interface between business sector and the public sector from various perspectives. And the question would be 'to what extent they have seen an increased interest from the business sector to ask these kind of questions or to ask for this type of information now that they can actually see what information is out there and how it can be used.'" + +I will put this to both groups when I meet them in Bratislava... The above point is seconded by Bogdan Manolea, who talks about how useful steps forward in e-procurement had previously been made when the data used to be published as a feed. Bogdan says he knew someone planning to make a business out of that data, rehashing it and selling it on, but then came the CAPTCHA codes which made automatic harvest impossible... + +> *Elena Calistru*: "**Datasets related to party financing**" Another feature of Bani Pierduti is that we make analyses. I'm now working on an analysis of party finance during the last election campaign. And sure, I checked the dataset available [...] I also made some requests for information from the Parliamentary Parties, but besides that, I took a camera [... and] took pictures of all the banners and so on, and afterwards sent a request for some prices from 3 companies offering outdoor media. The differences are at least significant. [...] the electoral financing laws says that no party can receive discounts or preferential rates for something, because that is considered a donation and they have to declare all the donations. And if one party will yell me "yeah, I got a discount" - why didn't you declare it? + +## Civil Society Collaboration + +My favourite part of the whole day is when the group start talking about how they might be able to help each other out by sharing the data that they each have added value to in some way. I ask: *"Do any of your projects archive the data so that it is available to the public for many years to come?"*. These are the responses we get: + +

Romania - Civil Society Collaboration from Open Knowledge Foundation on Vimeo.

+ +(*NOTE: The Soros Foundation reports are already online, not in open formats, but a little bird tells me it is something that is being considered for future*.) + +Safe to say, I think it would be a great start if more Civil Society Organisations shared data between themselves as well. + +## More to come + +This is just a tiny sample of a great and rich debate, I haven't even touched on here the views of the group on whether social media is a hindrance or a help for their work, other interesting methods employed to get participation, both from active citizens and governments in the decision-making process. diff --git a/examples/openspending/content/blog/2012-07-27-budapest-institute.md b/examples/openspending/content/blog/2012-07-27-budapest-institute.md new file mode 100644 index 00000000..05ada3d9 --- /dev/null +++ b/examples/openspending/content/blog/2012-07-27-budapest-institute.md @@ -0,0 +1,94 @@ +--- +authors: +- lucy +redirect_from: /2012/07/budapest-institute/ +title: Athens to Berlin - PROFILE - Our Money, the Budapest Institute +--- + +**This post was written based on the contributions of Petra Reszkető, Balázs Váradi and Timea Sütő of the [Budapest Institute](http://www.budapestinstitute.eu/), Hungary. The video of their interview will be displayed as part of the complete series [Athens to Berlin](http://openspending.org/blog/2012/07/05/OSI.html).** + +*The Budapest Institute is an independent think-thank. It produces public policy research and analyses to support policy-making in Hungary and in the Central-Eastern-European region. It is part of our mission to promote evidence-based policy-making and to try to make the national public policy discourse based more on facts rather then beliefs. This post is about the project 'A Mi Penzünk' which aims to present the budget spending of the Hungarian State simply and interactively to make it engaging for the interested layperson.* + +[A Mi Penzünk's](http://amipenzunk.hu/) aim is to present financial data for the Hungarian State broken down by functions to provide information on what exactly tax money is spent on. With the visual presentation (thanks to the open software developed by the [Open Knowledge Foundation](http://okfn.org/)) of the budgetary expenses and with the database that can be mined and freely downloaded from the website. + +## What is the aim? + +Main policy goals of the project are the following: + +* to make ’dry numbers’ digestible and illustrative by visualizing the Hungarian budget (spending lines) +* to dispel misbeliefs and popular fallacies on spending priorities of Hungarian governments by sharing information and providing policy narratives. + +With this project we would like to contribute to the national tax consciousness and to the strengthening of civic responsibility. Our partners were the Open Knowledge Foundation (visualization software) and the Fiscal Responsibility Institute Budapest (converting and editing the database). The Budapest Institute for Policy Analysis started the project in June 2011, as a revival of an earlier, Hungarian initiative with the same name. + + + +## Who is it targeted at? + +First, we aim at informing young people (high school students) who are just now growing up to become tax paying citizens by orienting secondary school teachers with the tools and applications installed on the website. + +For students we drew up “homework” exercises, and we also created visual aids, posters and other teaching materials to support the work of interested high school teachers. For journalists we provide some short essays introducing policy narratives on the raw budget data. + +Second, we would like to facilitate the work of journalists who may use this kind of information and data. One of the conclusions of the workshop with journalists and the following consultations with media representatives was that our initial idea turns to be a false hope. The level of data journalism is far below the EU average in Hungary and in addition, the Hungarian media actors/ firms can not fit this rather seasonal content into their business model. We have learned that the Hungarian media is prone to get ready-made analytics and reports rather than to perform investigative projects on its own. + +## Why is it necessary? + +In Hungary, policy debates and the media frequently discuss government expenditures. But factual, politically unbiased and easy-to-understand information about actual numbers and proportions is in short supply. Both electioneering and run-of-the-mill political communication is, to a large extent, about how much should be spent on schools and pensions, trains and healthcare. For the average citizen these statements are hard to interpret without points of reference. Are those sums too much or too little? Compared to what? What else could we spend the billions potentially saved on? During our work we have often had to face up to the fact that there is no convenient public database that would represent the budget of the Hungarian state. True, we are informed about the general budget from the current year’s budget law and, in the year after, the law on the final accounts. It is, however, quite a challenge to learn about the planned and actual costs and revenues broken down by the functions of government, not institutions, in a unified and transparent way. + +Usually if a tax-conscious Hungarian citizen wants to gather information on the central budget he or she can only do so upon patiently waiting for years, accessing the Central Statistical Office’s or the Eurostat’s homepage. There is unfortunately no public database available on the website of the Hungarian Government. + +With this project not only that we provide up to date, accurate information organised in a downloadable database, but we present the data broken down in such way, that it becomes clear and easily understandable on which functional activities is the taxmoney spent (cf. COFOG standards), not - as it was customary before - which institutions receive it. + +## How did you get the data? + +At the very start of the project we faced several challenges relating to accessing the official data. We struggled with the National Statistical Office and the Ministry of National Development for months, we finally had to ask for support of a later partner organization, the [Fiscal Responsibility Institute Budapest](http://www.kfib.hu/) in July 2011. After an overlong awaiting for the data we cleaned and restructured the database according to our needs, and built the website. + +## Features + +The project: + +* developed an online public information resource presenting information on public spending, thereby matching a niche due to the lack of availability of any official public database +* applied and presented visualization software that make raw data easy to digest +* contains two online [posters](http://amipenzunk.hu/front/poszter) downloadable from the website, one available in 50 printed A1 copies +* Contains a feature called ['Your Dream Budget'](http://amipenzunk.hu/front/dream?country=AT) presenting the relationship between different expense items and 5 international comparisons, allowing you to assess the spending priorities of your government in comparision to other countries. +* Entailed one media workshop, and two workshops with teachers and representatives of educational organizations +* Has social media profiles ([Facebook](https://www.facebook.com/pages/amipenzunkhu/186571038102340), [Twitter](https://twitter.com/amipenzunk).) + + + +## What was the impact? + +In all this we have relied a great deal on the input of the participants of the three workshops we organized for teachers and journalists, and on the recommendations of peer colleagues. + +The workshops have proved that such projects can grab attention, and there is explicit need for follow up. However we have also learned that, contrary to our expectations, the real interest for our project is not shown by the media, but by teachers and NGOs with an educational profile. + +For the time of being, we are in consultation with workshop participants from high schools and educational organizations (NGOs) and we are still receiving orders for the poster. + +The long term impact of this initiative can hardly be measured in a proper way, though the real immediate impact we hope can be demonstrated later by the webpage statistics (visitors, users of applications, downloads of data and poster) and by the steady demand for the poster to be ordered at re-production costs. [As of April 2012], since the website was officially published in mid January we have had over 10000 visitors out of which 16.66% are returning ones. 87.84% of our visitors are from Hungary, mainly from Budapest and the bigger cities. + +## Where Next? + +Based on the workshops and follow up consultations with teachers and educational experts, we drew the conclusion, that there is a real need: + +* for developing a comprehensive teaching modul (more teaching material explaining terms, clarifying definitions, class schedules, etc.) +* for extending the project to the revenue side (e.g. visualization, trade-off between revenues/taxes and expenditures, other interactive applications highlighting balance at the individual/ household level) +* for creating English version of the website and providing more and deeper international analysis +* to update the database with fresh data for 2012, to create the English version of the website, and to develop another version of the webpage that would be accessible for people with disabilities. + +## What could be done to make your work easier? + +Based on our experience implementing this project the BI is interested in organising and/or participating in workshops focusing on issues like: + +* working in an interdisciplinary scheme (developer-desinger-economist) +* developing applications and interactive tools – how to test them? +* enhancing usability / promoting techniques + +Datawise, we also need data classified by function (i.e. programme)! + +*Stay tuned for more updates from the [Athens to Berlin series](http://openspending.org/blog/2012/07/05/OSI.html) via the [OpenSpending Mailing List](http://openspending.org/blog/2012/07/27/budapest-institute.html).* + + + + + + + diff --git a/examples/openspending/content/blog/2012-08-05-introduction-to-the-taxman.md b/examples/openspending/content/blog/2012-08-05-introduction-to-the-taxman.md new file mode 100644 index 00000000..8f440cf8 --- /dev/null +++ b/examples/openspending/content/blog/2012-08-05-introduction-to-the-taxman.md @@ -0,0 +1,125 @@ +--- +authors: +- nick +redirect_from: /2012/08/introduction-to-the-taxman/ +title: An introduction to the TaxMan +--- + +One of the pieces of technology powering [Where Does My Money Go][wdmmg] is a standalone application, TaxMan, that performs a rather dull yet important task. As anyone who has filled out a tax return will know, tax can be rather complicated, with numerous steps and calculations to perform. If you are lucky, some of these calculations may be performed for you by your national revenue agency. This however doesn't help you if your aim is not only to calculate *your* tax, but also to understand how taxes are calculated in general. With [Where Does My Money Go][wdmmg], we wanted to find a way to estimate the total tax payments of UK citizens based on their salaries, income tax and "indirect" payments such as VAT (sales tax), fuel duties, etc. + +[wdmmg]: http://wheredoesmymoneygo.org/ + +Enter the TaxMan. +TaxMan is a really simple JSON-over-HTTP API that aims to provide current and historical tax calculators for jurisdictions around the world. It currently has support for South African, Mexican, and British tax codes, including estimated calculations for British indirect taxes, and it's easy to extend to other countries. Critically, it doesn't try to shoehorn a complicated algorithmic tax code into a tabular format. + +Before we go into details about how TaxMan works, let's see what it does. For example, find out which jurisdictions are supported by TaxMan (the following examples use [HTTPie][httpie] on the command line, but you can use anything capable of making HTTP GET requests, such as your browser or jQuery): + + $ http GET taxman.openspending.org + HTTP/1.1 200 OK + Content-Type: application/json; charset=utf-8 + Content-Length: 75 + Date: Wed, 01 Aug 2012 07:42:31 GMT + + { + "jurisdictions": { + "gb": "/gb", + "za": "/za" + }, + "message": "Welcome to the TaxMan" + } + +We explicitly link to the jurisdictions from the root of the API so that any client library can refer to `jurisdictions.gb` rather than having to "hard code" any of TaxMan's URL structure. So let's +follow the link for South Africa, here shown as `za` (TaxMan by convention uses [ISO3166 codes][iso3166] to denote countries): + +[iso3166]: http://www.iso.org/iso/country_codes.htm + + $ http GET taxman.openspending.org/za + HTTP/1.1 200 OK + Content-Type: application/json; charset=utf-8 + Content-Length: 331 + Date: Wed, 01 Aug 2012 07:47:31 GMT + + { + "calculation": { + "total": 0 + }, + "data": { + "income_tax": { + "bands": [ + { + "rate": 0.18, + "width": 160000 + }, + ..., + { + "rate": 0.38, + "width": 133000 + }, + { + "rate": 0.4 + } + ] + }, + "rebates": { + "aged_65_to_74": 6390, + "aged_75_plus": 2130, + "base": 11440 + } + }, + "options": { + "age": null, + "income": null, + "year": 2012 + } + } + +There's plenty to take in here, so let's focus on the basic structure first. There are three top-level keys in the response: `calculation`, `data`, and `options`. As you might expect, `calculation` contains the results of any tax calculation performed by TaxMan. In this example, no calculation has been performed, as we didn't give TaxMan an income to use for the calculation. Thus there's no interesting data. TaxMan will however also attempt to provide the data *it used to perform its calculations*, which is shown in the `data` field. The `options` field makes explicit the available options for this calculator. Note +that we can supply an `income` and an `age`, and some of the entries in `data` begin to make more sense. The `data.income_tax.bands` key contains a description of South Africa's tax bands. All bands have a tax `rate`, and all but the last band has a `width`, denoting the width of the band of income taxed at that rate. The last band covers all higher income, so has an effectively infinite width. For example, a tax system which charges 10% tax on all income up to £40,000, and 20% thereafter, would have two tax bands: + + [{ "rate": 0.1, "width": 40000 }, { "rate": 0.2 }] + +So, what happens if we do supply an income? I'll truncate the `data` section of TaxMan's output for the sake of clarity. + + $ http GET 'taxman.openspending.org/za?income=200000' + HTTP/1.1 200 OK + Content-Type: application/json; charset=utf-8 + Content-Length: 468 + Date: Wed, 01 Aug 2012 08:06:07 GMT + + { + "calculation": { + "income_tax": { + "bands": [ + 28800, + 10000, + 0, + 0, + 0, + 0 + ], + "total": 38800 + }, + "rebates": { + "age_related": 0, + "base": 11440, + "total": 11440 + }, + "taxable": 200000, + "total": 27360 + }, + "data": { ... }, + "options": { + "age": null, + "income": 200000, + "year": 2012 + } + } + +You'll notice that the `calculation` section now contains much more information about the calculation, including data on how much you paid in each tax band, your tax rebates, and the total payable tax (`calculation.tax`). + +And that's really all there is to TaxMan. It enforces few of the conventions mentioned above on its calculators, although I hope that the `calculation`, `data`, `options` triptych will become a pattern throughout. Most importantly of all, TaxMan is intended to be a simple and discoverable API. Explicitly representing available options is part of this aim of "discoverability". + +Under the hood, TaxMan is a very simple open-source [Node][node] application. Calculators are typically one or two files. You can [find the source code on GitHub][tmgh], and we encourage you to fork it and add a calculator for another jurisdiction -- perhaps your own -- or improve an existing one. + +[node]: http://nodejs.org/ +[tmgh]: https://github.com/openspending/taxman/ diff --git a/examples/openspending/content/blog/2012-08-15-Lost-Money.mdown b/examples/openspending/content/blog/2012-08-15-Lost-Money.mdown new file mode 100644 index 00000000..db38147b --- /dev/null +++ b/examples/openspending/content/blog/2012-08-15-Lost-Money.mdown @@ -0,0 +1,98 @@ +--- +authors: +- lucy +redirect_from: /2012/08/Lost-Money/ +title: Athens to Berlin - PROFILE - Bani pierduti? (Lost money?) +--- + +This is a profile of a very interesting new project coming out of Romania, aiming to make government finances understandable for the average citizen. It is written based on contributions from [Elena Calistru](https://twitter.com/MadamadePica), who kicked off the project. + +# Vital Statistics + +* Name of Project: Bani pierduti? (Or, in English, "Lost money?") +* Link to project: +* Approximate number of users engaged through the project: >30.000 + + + +## What is the background of the project? + +The project is one of the five winners of the [Restart Romania 2011](http://restartromania.netsquared.org) competition, initiated by [Techsoup Romania](http://www.techsoup.ro/) with the support of the US Embassy to Bucharest. + +Starting at the beginning of August 2011, 104 projects were registered for the Social Justice Challenge Restart Romania, and went under the scrutiny of the community. In the end, a jury formed by representatives of the diplomatic community, business sector and IT industry decided the selection of 10 finalist projects. Between 28 - 30 October, the Restart Romania Hackathon transformed the ten ideas with the help of programmers and communication specialists into more concrete platforms which were presented within the Restart Romania Gala. Bani Pierduti was voted within the Gala as one of the five winners of 5000 USD funding. + +## What are the aims? + +The project formerly known as *"Where’s my LEI, man?"* entered the competition aiming to centralize the publicly available financial information regarding the projects financed through public money (budgets, annual reports etc.).The main objective was make authorities accountable in the manner in which public funds are spent. + +After winning the Restart Romania Gala, the project went through a consistent re-thinking aiming to identify both the best technologies for a more complex platform than initially planned, and the necessary data sets which allowed the best representation on how public funds are spent in Romania. Thus, if at the very beginning the project only aimed to use the state budget data, it now operates with data comprising the budgets dedicated to social assistance and public health, the budgets at local level for the Romanian counties, projects financed through EU funds, comparisons with the percentage allocated to various sectors in other EU counties etc. + +The project is a now a permanent programme of a newly-established NGO – [Funky Citizens](http://www.funkycitizens.org/) (website under construction at time of publication). Which aims to engage civil society (taxpayers) in the decision making process related to public funds through the use of technology. Its major objectives are: + +* Quantitative and qualitative growth of the awareness on the issue +* Offering information and tools for influencing the decision-making process +To achieve its objectives, the project relies on three pillars: +* Data & process presentation +* Public participation +* Understanding the bigger picture + +## How does the platform tackle the issues you outlined? + +The three pillars of the platform respond to the following problems: + +Problem #1: Fiscal policies represent a "mystery" for the majority of citizens +

Consequences: Lack of information and understanding of the process; scarce public oversight of public funds administration; public spending is associated with corruption and distrust

+

How we respond: Educate citizens on the topic

+ +Problem #2: Little or no participation of the community to the fiscal policy +

Consequences: Limited use of existing tools for participation to the decision-making process; needs of the community not reflected in the resource allocation; no feedback to the policy makers on their decision

+

How we respond: Facilitate direct participation

+ +Problem #3: Lack of vision from governments on investment/ development priorities +

Consequences: Short-term planning leading to limited predictability and accountability; bad administration, mismanagement, or corruption in public spending; incoherence between the fiscal policy and other public policies

+

How we respond: Analyse and understand data

+ + + +## What is the role of technology in the approach to solving that problem? + +The role of technology is an important one, since the web-based platform is the main feature of the project. So far, transparency in the fiscal policy can be achieved only through complicated documents published on the websites of the authorities or through FOIA requests. Also, there were no e-participatory budgeting experiences so far, the only manner to organize public debates on budgetary issues being offline events. + +## What are the successes of this project? + +The project is still very young and in its early stages. However, the evaluation of its outcomes already shows several approaches which proved successful: + +* A consultation process with relevant governmental stakeholders prior to the launch of the project proved to be a good approach in ensuring a supportive or at least a not contentious interaction with the authorities, given the sensitivity of the subject. +* The gradual implementation and launch of the features of the platform seems a successful strategy to educate citizens on a difficult subject while creating interest and awareness on the topic. +* The engagement of different categories of supporters of the project (from young dynamic professionals to the diplomatic community) ensured a greater visibility for the initiative and is expected to further enlarge the community of advocates for more transparency in fiscal matters. + +## Are there areas where the project failed? What are the challenges? + +The main challenges of the project are mostly related to two major issues encountered by such initiatives: + +* The absence of an open data approach in the release of official information related to public spending makes the implementation of the project slower as well as resource-consuming. +* A general perception that public money are lost due to corruption makes people less inclined to look closer at the entire policy cycle and thus the efforts to educate or to engage them harder. + + + +## Have you had particular problems with the data? + +Even though Romania has just joined the Open Government Partnership, the implementation of the open data format for governmental data sets is expected to take at least a few years. Thus, the various data formats present on the websites of authorities or even their absence in several cases made data collection a rather difficult process. + +## Are you actively seeking the involvement of the user groups? + +The project also envisages that an entire pillar of the platform (“public participation”) will actively seek the involvement of the user groups. The implementation of this service started with two features (large investment projects timelines and legislative early-warnings) which seek an interaction with the public and future plans propose to increase the amount of citizen participation. For example, there are plans to do this by: + +* encouraging direct feedback into laws already in draft stages which allows users to cut, add to and restructure proposed to be bills on the basis of the desired budgetary outcome, +* building a simulator for the central budget - Allowing people to visualise and explore the effect of different to be +revenue and expenditure policies (e.g. raising taxes) +* promoting public participation in the annual budget cycle through a calendar of debates on budgets as well as pilot +offline events with webcasts + +The most consistent involvement features are expected to be implemented by the end of 2012 – early 2013, as a second stage in the development of the project. + +## What are the plans for the future? + +The project was planned as a continuously growing platform and its scaling or additional features were taken into consideration from the very beginning. A mobile feature is expected to be implemented into the web platform in 2013, a plan which also involves the use of social audits for public contracts. + +*Thanks to Elena for putting this post together and we hope that she will stay in touch with updates about the impact of the platform when it becomes more established. We are always looking for the most exciting case studies from around the world. If you know of one we should feature, please drop us a line via the [OpenSpending Mailing List](http://lists.okfn.org/mailman/listinfo/openspending). Interested in helping to make data open in Romania? Start a discussion on the [OKFN-Romania list](http://lists.okfn.org/mailman/listinfo/okfn-ro).* diff --git a/examples/openspending/content/blog/2012-09-11-Spending-Data-Handbook.mdown b/examples/openspending/content/blog/2012-09-11-Spending-Data-Handbook.mdown new file mode 100644 index 00000000..9f543e1e --- /dev/null +++ b/examples/openspending/content/blog/2012-09-11-Spending-Data-Handbook.mdown @@ -0,0 +1,79 @@ +--- +authors: +- lucy +redirect_from: /2012/09/Spending-Data-Handbook/ +title: Spending Data Handbook - How do Civil Society Organisations Wrangle Spending Data? +--- + +Over the last few months, we have been travelling the globe in an effort to work out [which organisations are mapping the ebbs and flows of government money](http://openspending.org/blog/2012/01/12/civil-society-and-spending-data-who-is-mapping-the-money.html) - and how they do it. In a series of interviews, currently being published via the [OpenSpending blog](http://openspending.org/blog/index.html) we've highlighted the issues which Civil Society Organisations face when working with government financial data. + +The aim has been firstly to establish a front along which Civil Society Organisations can unite and realise that they are not lone-voices demanding *better* data, *more current* data. + +# The story so far... + +Our first step was to start building a [Working Group](http://openspending.org/resources/wg/index.html) to bring these people who work in similar sectors around the globe together and stimulate discussion around the topic of Spending Data. There is already a good exchange of interesting projects from around the world and topics such as the creation of standards and an exchange of expectations and practices are being mooted. The group is growing in size and open to those working in the field of open spending data. + + + +The second aim was to find a way, via training, technology or otherwise, to tackle the challenges which the CSOs we spoke to highlighted. The most common problems that these organisations have been encountering include dodgy data, which often changes in structure and format from year to year, incorrect data formats (most common offender, as usual, being - PDFs), jargon or codes in data which had to be painstakingly decoded and many more. We were also delighted to see that CSOs were curious and getting more ambitious with data and wanted to know more about how they could work with it most effectively, for example asking questions such as- *"How do I present my data better?"*, *"How do I speed up getting data from websites?"* or *"We've been thinking about geocoding publicly-funded projects so we can put them on a map, do you know of anyone who has done this successfully?"*. + +# What's next? - The Spending Data Handbook. + +The range of advocacy topics tackled by these groups is so diverse (from gender budgeting to checking that promised infrastructure in a local town actually gets built) that it would be impossible to address all of the data wrangling skills in one book. Everyone needs different levels of data, from tiny, ward-level datasets up to national budgets. However, there are some overarching principles which apply universally to working with government financial data. + +These overarching topics are what we aim to cover in the Spending Data Handbook. Like the [Open Data Handbook](http://opendatahandbook.org/en/), it will be available as an open educational resource on the internet and for training sessions. More on the content below... + +# What questions will the Spending Data Handbook cover? +

+
+ Based on the interviews we conducted, we've drawn up a suggested list of topics for what could go into the handbook. We'd love to hear from you with your suggestions and modifications. +
+ +## Part One: An introduction to Open Financial Data + +

+ +### How can CSOs be more effective in their use of data? What should they be asking governments for? + +* How can they be more effective in requesting (and keeping hold of) meaningful information from government? +* Which phases of the budget/procurement cycle do they need to demand data from? +* What technical formats are ideal for re-use and interpretation? +* What transparency rules need to be in place to enforce publication? + +### How can they get (and keep hold of) data + +* How can they make backups of the data that has been published? +* How can they extract data from sources on the web? + +# Part Two: Technical primer for data work +

+### How can data be analysed and interpreted? +* Which phases of the budget/procurement cycle produce which kind of data? Which tools do you need to work with these different types of data? +* What different data formats can be used - and how? +* What is the difference between structured and unstructured digital data? +* How can unstructured data be re-structured? +* What are PDF files and how can information from them be extracted? +* How can you convert between different structured formats? + +### How can data be cleaned up and brought into a more uniform format? +* How can data be augmented to allow for more meaningful interpretation? +* How can government classifications (codesheets) be applied? +* How can geographic information be included? +* How can information about vendors/suppliers be included? + +### How can data be delivered to the public? +* What are databases, query languages? How can they be used? +* How can you summarize large sets of data? +* How can data be presented in an accessible and meaningful way? + +## Any thoughts? + +So now, **we need your feedback**. Are you an NGO working on these issues? Are there any additional topics you feel people in your organisation would like to know more about? Are the suggested topics useful? You can get in touch with us via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending) or submit a response [via this form](https://docs.google.com/spreadsheet/viewform?formkey=dEl6TFh4THpOVkg1SmZaZXcycUdyRWc6MQ). + +## When? + +We hope for a sprint on the Handbook to take place around November, but the book will be a work in progress. There are many issues raised by NGOs in our research which won't make it into this first version, but we hope that as organisations become more ambitious in working with their data, we'll add tips and tricks for advanced wrangling into future versions. + +*Further interviews from conversations with CSOs will continue to be published on the [OpenSpending blog](http://openspending.org/blog/index.html). Stay tuned for updates.* + + diff --git a/examples/openspending/content/blog/2012-09-13-uk25k-reporting.mdown b/examples/openspending/content/blog/2012-09-13-uk25k-reporting.mdown new file mode 100644 index 00000000..af0f8414 --- /dev/null +++ b/examples/openspending/content/blog/2012-09-13-uk25k-reporting.mdown @@ -0,0 +1,43 @@ +--- +authors: +- lucy +redirect_from: /2012/09/uk25k-reporting/ +title: UK Departmental Government Spending - Improving the Quality of Reporting +--- + +**Continuing in their mission to make spending data more accessible and comprehensible, the Spending Stories team and the team of Data.Gov.Uk are releasing a reporting tool today that will help journalists and analysts to pick the freshest and best departmental spending data to work with when exploring the UK central government expenditure.** + +### Spending data is juicy for journalists - why does it get neglected? + +Many reasons. One key one is that the shelf-life of a spending dataset is pretty short from a journalist's point of view, if they have to wait 6 months or even a year for spending data they need for a story to be released, then chances are - the sniff of the story they were wanting to write will probably have gone stale. + +Journalists, campaigners and activists need access to well-structured, machine readable and timely data from national as well as sub-national administrations. At OpenSpending, we're often contacted by journalists with story ideas, or they approach us with a lead. The stumbling stone for them is either lack of information, or worse data that they can't use because they are not sure of its completeness. The problem is thus the one of trees falling in a wood: If a transaction is missing from a list - does that mean there was no transaction for that amount on that date, or does it mean that the transaction simply was not reported? + +These distinctions are important for anyone trying to understand the data - and up to now they have been pretty tricky to answer. As an attempt to make this a little easier, today, we announce the availability of an automatic reporting tool for spending data (available both on [data.gov.uk](http://data.gov.uk/data/openspending-report/index) and on [OpenSpending](http://openspending.org/resources/gb-spending/index.html)), the result of a collaboration between data.gov.uk and us in order to increase the visibility of the spend data and to increase the ease of browsing the substantial volume of datasets that make up the [reporting of Government expenditure](http://data.gov.uk/openspending) in data.gov.uk. + +The [tool lists departments](http://data.gov.uk/data/openspending-report/index) registered as data publishers on data.gov.uk and details how precisely they have followed the [HM Treasury reporting guidelines](http://www.hm-treasury.gov.uk/psr_transparency_index.htm). It will also make the whole of the reported data available for search and analysis both on [data.gov.uk](http://data.gov.uk/openspending) and on [the OpenSpending site](http://openspending.org/search). + + + +The tool is useful to those both using the data, and those within government in assuring that departments are reporting on time. It helps to check: + +1. Quality of the data (i.e. adherence to HMT reporting guidelines, well-structured data) +2. Status of reporting (i.e. how complete the reports are or if there is a reporting period missing) + +### Why was this possible? + +Having all of these datasets organised under a single catalogue at Data.Gov.UK  in simple spreadsheet format combined with the Data.gov.uk team's work in making the necessary metadata available enabled the OpenSpending team to create an extraction system to be set up to clean the data on a regular basis. The team then cleaned over 6000 column names to add compliance with [HMT guidance](http://nomenklatura.okfnlabs.org/uk25k-column-names). + +### How does it work? + +The report generator then highlights in red departments who are registered as a publisher on Data.gov.uk but have failed to publish any information on their spending, in yellow those who have published data which cannot be interpreted as spending data (e.g. PDF format or not complying with [the template](http://www.hm-treasury.gov.uk/d/transparency_annexa100910.xls) provided by HMT) and green those departments whose records have been updated as regularly as demanded as per the publication requirements (latest data must have been published as recently as a month ago). + + + +The first stage of this release deals with central departments, who are obliged to report all spending over 25,000 GBP. Subsequent stages to follow soon after will monitor local councils and other governmental bodies, which have different reporting requirements. The interface will be useful both inside and out of government, to ensure transparency regulations are met and to better understand where gaps in data may alter the completeness of the picture offered by government data. + +* Reporting tool on [data.gov.uk](http://data.gov.uk/data/openspending-report/index) +* More extensive list of entities at [OpenSpending](http://openspending.org/resources/gb-spending/report/index.html) +* [Our briefing](http://openspending.org/resources/gb-spending/index.html) with all the details and link to code. + +*Interested in more regular updates from the Spending Stories team? Join the discussion via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending).* diff --git a/examples/openspending/content/blog/2012-09-17-participatory-budgeting-finland.md b/examples/openspending/content/blog/2012-09-17-participatory-budgeting-finland.md new file mode 100644 index 00000000..9d3f67d0 --- /dev/null +++ b/examples/openspending/content/blog/2012-09-17-participatory-budgeting-finland.md @@ -0,0 +1,39 @@ +--- +authors: +- raimo +redirect_from: /2012/09/participatory-budgeting-finland/ +title: First steps for Participatory Budgeting in Finland +--- + +**This post is written by [Raimo Muurinen](https://twitter.com/ra__mu) and [Henri Ahti](https://twitter.com/HenriAhti) who are taking a lead on one of Finland’s first Participatory Budgeting (PB) projects with Helsinki City Library. The post comes just as we prepare for Tiago Peixoto’s (aka [@participatory](https://twitter.com/participatory)) keynote and talk on [how to involve citizens in the budget making process](http://okfestival.org/participatory-budgeting/) at OKFestival.** + +## How did the project get going? + +In April 2012, the Finnish innovation fund [Sitra](http://www.sitra.fi/en) organized [the first participatory budgeting process in Finland](http://www.sitra.fi/artikkelit/2012/osallistuva-budjetointi). As a result, among few others, a new PB project was elected to receive funding. [Helsinki City Library](http://www.lib.hel.fi/en-GB/) users now have the chance to plan and decide projects for the upcoming new [central library](http://keskustakirjasto.fi/en/). + +## Why is this necessary and why should it happen now? + +At the end of 2011, the [Democracy Group of the City of Helsinki](http://demokratia.hel.fi/english), led by mayor Jussi Pajunen, published a report saying PB has not been used previously in Finland. Except for, with a broad understanding, the area of the northernmost city of Finland, Rovaniemi, where regional boards have a strong role in budgeting. + + + + +At the same time Sitra, a Finnish innovation fund, took a more practical approach to the matter. Between August 2011 and April 2012 they organized, the [New Democracy Forum](http://www.sitra.fi/en/new-democracy), which examined and sought for a 'new type of democracy' with 30 guest participants invited from different fields of expertise. A varied group of journalists, members of the parliament, public administration officials, SME CEOs and citizen activists among others heard about the concept of participatory budgeting and as a result, Sitra decided to organize a small-scale experiment, as it had budgeted ca. 100,000 euros for the arising new democracy initiatives. 16 proposals for new democracy projects qualified for voting, where the forum participants, spokespeople for the projects and a few employees from Sitra had each one vote per person. The voters were to vote for a list of ten preferable projects to receive funding. Prior to voting, all the projects’ spokespeople had pitched their projects. As a result, [Avoin ministeriö](http://www.avoinministerio.fi/) (Open Ministry), [Sosiaalinen hub](http://sosiaalinenhub.wordpress.com/) (Social Hub) and our Participatory Budgeting proposition for the Central Library were voted to receive funding. + +## Who will be involved in implementing it? + +Behind the PB proposal were companies [Emobit Oy](http://emobit.fi/en) and [Avanto insight Oy](http://avanto.in/) as a partner and the Helsinki City Library as a client. The plan started with gathering information about PB and setting up a free and open resource site [osallistuvabudjetointi.fi](http://osallistuvabudjetointi.fi/), which is now running the early first version. + +Emobit leads the project as contractual partner for Sitra, providing technical expertise for [OpenSpending](http://openspending.org/), which is used to visualize library spending from previous years with data from [Helsinki Region Infoshare](http://www.hri.fi/en/). + +Avanto insight is responsible for planning the PB processess and is running the osallistuvabudjetointi.fi site. The company is a startup founded in 2011 and based in Tampere. + + + +## How will it work? + +Focus is on allowing library users to participate on developing the new Central Library, which is currently waiting for the results of the architectural competition. At the moment, we are planning to hold 2-3 live sessions in different libraries. These will be open, creative participation sessions for anybody interested. The sessions will give people the possibility to create their own budget proposal and also draw on feedback from the [Dream! survey](http://keskustakirjasto.fi/en/2012/06/07/dream-the-central-library-to-reality-weekend-at-the-pavilion-16-17-june/) held at the beginning of 2012, where Helsinki residents were invited to design the type of urban space and culture they would like to see blossom in the library. Planning is conducted openly, as we publish all the documents freshly to the osallistuvabudjetointi.fi site, where it is possible to comment them and affect the execution. + +[View the data on OpenSpending](http://openspending.org/helsinki_city_library_2009_2011_1) + +*We’re looking forward to seeing some pictures from the living results of these sessions. Look out for Raimo, Henri and Tiago on Friday, 21st September at OKFestival. We hope that bringing these two groups together will spark a great discussion on the future of PB in Finland.* diff --git a/examples/openspending/content/blog/2012-09-24-OKFest-Followup.md b/examples/openspending/content/blog/2012-09-24-OKFest-Followup.md new file mode 100644 index 00000000..83b657c4 --- /dev/null +++ b/examples/openspending/content/blog/2012-09-24-OKFest-Followup.md @@ -0,0 +1,60 @@ +--- +authors: +- lucy +redirect_from: /2012/09/OKFest-Followup/ +title: Where Does the Money Go - International. Updates from OKFestival +--- + +We got a chance to catch up with many old OpenSpending friends at [OKFestival](http://okfestival.org/) last week and got the chance to meet some new ones. For any of you who missed the action, here's a quick recap... + +## Machine Readable Budgets for the Netherlands + +The week started with Ton Zijlstra announcing a scoop in the "State of Play: The Open Government Data movement and related initiatives", for the first time, the Netherlands announced that it would publish its budget plan in machine-readable format. + +Naturally, we wanted to get this into OpenSpending as quickly as possible... but when working with the data, we had some questions. For example, we could not make the numbers align with any other [official estimates](http://rijksbegroting.nl/binaries/pdfs/miljoenennota_bijlagen.pdf) (see page 5). With the help of representatives from the Court of Audit of the Netherlands we did a comparison of the differences of the [open format](http://opendata.rijksbegroting.nl/opendata.html) with the [PDF version](http://rijksbegroting.nl/binaries/pdfs/miljoenennota_bijlagen.pdf) and could not make all of the numbers match (PDFs as reference material can be useful!). We've approached the Ministry of Finance to clarify the discrepancies between the two documents and hope to have an update soon. + + + +## Converging on Standards + +Publishing to the IATI standard is now becoming more commonplace, but how do we take it to the next level? The OpenSpending team is particularly keen to work towards [standards for government financial data](http://openspending.org/resources/standard/index.html) - so that we can answer questions such as 'how do the budgeting priorities of governments in developing countries align with donor priorities?' (as we did for Uganda with our project with Publish What You Fund) on a systematic and automated, rather than a time-consuming case-by-case basis. Such standards may make other, more ambitious projects, possible at scale, such as geocoding projects and mapping the budgeting process from start to finish, e.g. to answer questions such as whether what was budgeted aligns with what was actually spent. + +In the session ["Open Development and aid flow: using open aid data"](http://okfestival.org/open-development-and-aid-flows-how-to-use-aid-data/) we discussed how important it is to align the spheres of budgeting and aid distribution and talked about some of the shortcomings and benefits of some of the systems currently in place. A few notes from the session: + + + +* First and foremost, we need to work out what can be combined and what can't, then produce the tools to map one to another. In order to do this, there is a **need to distinguish between file formats (*grammar*) and reference data (*vocabulary*)** if we are interested in making interoperable standards. +* While IATI is by and large considered a success in terms of getting aid data published, there are still some areas for improvement. For example, **how can we spot which data is missing in order to know how complete a picture we have of total aid to developing countries?** To a certain extent, this has been done in the UK for spending data with the recent announcement of the [report generator](http://openspending.org/blog/2012/09/13/uk25k-reporting.html) for Data.Gov.UK. The question is: *could a similar tool be built for the IATI registry?* +* Feedback on data. Even within those organisations who publish their data to the IATI standard - there are discrepancies in what data is contained within. For example, some publishers may leave fields blank while completing all other fields. It was felt within the group that **there should be some kind of feedback mechanism** to encourage publishers to improve the quality of the data that they published. +* Lastly - **the forum for this discussion needs to be decided**. Should bringing these people together be a role of the [Open Government Partnership](http://www.opengovpartnership.org/), or of the [Global Initiative for Fiscal Transparency](http://fiscaltransparency.net/) or do we need a specific forum for this? + +We're going to be continuing this discussion over the coming weeks via the [OpenSpending Mailing list](http://lists.okfn.org/mailman/listinfo/openspending). We welcome input from anyone involved in this area, so please, join the mailing list and chime in. + +## Connecting with Citizens + + + +We were very fortunate to have a fantastic panel of speakers from around the world to talk about their experiences in connecting government financial data to citizens' issues in the panel ["The Money and the Many"](http://okfestival.org/the-money-and-the-many/). Our stellar cast included: + +* Damir Mehmedbasic, Executive Director, [Public Interest Advocacy Center (PIAC)](http://www.cpi.ba/). Bosnia and Herzegovina. +* Oluseun Onigbinde, Team Lead, [BudgIT](http://yourbudgit.com/). Nigeria. +* Federico Ramírez Corona, Lead Programmer, Fundar, Centro de Análisis e Investigación. Mexico. Talking about Fundar's [Farm Subsidies Project](http://subsidiosalcampo.org.mx/index.html/) - blogpost to follow. +* Gisele Craveiro, Professor and Coordinator of Research Group on Public Policy for Access to Information (GPOPAI), University of São Paulo. Brazil. Talking about the ["Caring for my Neighbourhood Project"](http://www.gpopai.usp.br/cuidando/). + +We heard a variety of approaches, from installing physical counters of the national budget in one of the busiest streets in Sarajevo, to actively engaging with twitter users and teaming up with the Occupy movement in Nigeria, to painstakingly geo-coding public projects in Brazil. + +Many NGOs around the world continue to grapple with this incredibly complex topic, and there were words of wisdom for those embarking upon the journey. Their tips: + +* The key to reaching the many is packaging media-compatible messages with specific budget information. +* It is important to bear in mind what type of audience you are writing for. If you are a team of academics, make sure you get someone on board who can help you translate the key issues into non-jargonised language for a layman audience. +* Online data-vis can be a key tool in delivering information, but it's not the end, (particularly in developing countries) you need tools to help get the message across offline, perhaps through wall paintings or mobile outreach. + +The topic of engagement with citizen issues is something which we will continue to discuss via the [OpenSpending Mailing list](http://lists.okfn.org/mailman/listinfo/openspending) and we look forward to hearing updates from teams like these all around the world on this topic. + +## Public Participation + +Starting with a bold opening statement? Why did Athens outperform other city states? - Better than usual information processing, leveraged by participatory institutions. Want to know how higher levels can lead to lower rates of child mortality? How participation can increase willingness of citizens to pay their taxes or improve the precision of budgeting practices? Watch the presentation below to hear Tiago's talk (from 45:00. It's worth it, there are kittens!). + + + +*If anyone does have photos or videos from any of the sessions, please upload them to Flickr and tag them with '#OKFest'!* diff --git a/examples/openspending/content/blog/2012-09-26-Balkan-Budgets.md b/examples/openspending/content/blog/2012-09-26-Balkan-Budgets.md new file mode 100644 index 00000000..939110dd --- /dev/null +++ b/examples/openspending/content/blog/2012-09-26-Balkan-Budgets.md @@ -0,0 +1,40 @@ +--- +authors: +- lucy +redirect_from: /2012/09/Balkan-Budgets/ +title: Balkan Budget Bubbles +--- + +**We're pleased to announce that the Where Does My Money Go bubbles are on their way to the Balkans! Thanks to a grant from the National Endowment for Democracy, the OpenSpending team will be working with the [Centre for Public Interest Advocacy](http://cpi.ba/) in Bosnia to produce interactive graphics of national and subnational budgets in Bosnia and lead training workshops with other Balkan countries (already on the guest list is [CRTA - Serbia](http://www.crta.rs/wp/en/)). We hope that this project will help organisations working in this area to form a united front to demand ever better data from governments, and to do ever more ambitious work with it.** + +CPI have already been very active in bringing the budget to the citizens of Sarajevo, particularly by having a strong working relationship with the media. They have also installed a live budget counter on the outside of their offices in one of the busiest streets of Sarajevo to bring information directly to those passing by. + + + + + + +
Live updates on government spending brought to passers-by in Sarajevo.
+ +

+ +CPI have already done a fantastic job of collecting the data, but there is still a lot of work to do to wrangle it into the necessary formats before it can be presented in the OpenSpending interactive graphics. + +## What will the project involve? + +As part of this project we will be: + +* Building visualisations of national, entity-, cantonal- and district-level budgets for Bosnia +* Doing training and capacity building workshops with organisations from other Balkan countries, on getting, wrangling and presenting financial data (with OpenSpending and other tools) +* Building a tax calculator for Bosnia (similar to the [Daily Bread](http://wheredoesmymoneygo.org/)), plus a session at the workshop on how to create one for other countries +* Test-drive the [Spending Data Handbook](http://openspending.org/blog/2012/09/11/Spending-Data-Handbook.html) and augment it with a more technical primer, for those organisations wishing to do more ambitious data work + + + + +
Damir Mehmedbasic speaks in Helsinki about CPI's work
+ +

+ +*As we progress with this project, we'll be focussing in depth on Balkan countries to work out which organisations are working in this area and what their technical requirements are. Know someone we should be in touch with? [Let us know!](mailto:info@openspending.org).* + diff --git a/examples/openspending/content/blog/2012-10-02-publicidad-oficial.md b/examples/openspending/content/blog/2012-10-02-publicidad-oficial.md new file mode 100644 index 00000000..4acf9914 --- /dev/null +++ b/examples/openspending/content/blog/2012-10-02-publicidad-oficial.md @@ -0,0 +1,138 @@ +--- +authors: +- federico +redirect_from: /2012/10/publicidad-oficial/ +title: PROFILE - 'Publicidad Oficial' - Official Advertising, Fundar, Mexico +--- + +**At Open Knowledge Festival, Federico Ramírez from Mexico presented Fundar's project on uncovering which government parties were using taxpayers's money to +finance their PR and advertising. In this post, based on an interview [Velichka Dimitrova](https://twitter.com/vndimitrova) did when she went out to Mexico for [OpenDataMx](http://blog.okfn.org/2012/09/04/opendatamx/), the Fundar team have delved deeper to talk about how this project came about, and hopefully inspire other organizations to tackle the issue in other countries.** + +> “Government advertising should be understood as a communication channel between the government and citizens. It should be clear, objective, easy to understand, necessary, useful and relevant to the public. It should not promote, explicitly or +implicitly, the interests of any party or government”. [*] + +## Vital Facts + +* *URL*: +* *Country*: Mexico +* *Scope*: Federal and state budget +* *Who*: Campaign implemented by [Fundar](http://fundar.org.mx/), [Article 19](http://www.article19.org/ ) and the Open Society Justice Initiative. +* *Contact*: [Justine Dupuys](mailto:justine@fundar.org.mx) (Fundar) + + + +## What is the Background of the Project? + +*'Publicidad Oficial'* is the expenditure of the government on public advertising. Fundar was particularly concerned that a lot of money in Mexico was being spent in promoting its own work and own image through public advertising, and aimed to tackle this. In Latin America, government advertising is often contentious: the relationship between the government and the media has frequently come under scrutiny. The amount spent on government advertising is very high in Mexico; in 2011, about 5027 million Mexican pesos / 385 million US dollars was spent, 75% of which went directly to television and radio broadcasting. During the past 12 years in Mexico there has been a huge discussion on how to take public money out of private (media) hands. + +In 2007, the Mexican constitution was changed to incorporate a ban on government advertising during political campaigns (Art.41) and also a ban on public servants (especially key executive officers, including thePresident, state governors, and municipal authorities) from appearing on +official publicity campaigns (Art. 134). The reasons for the reform included the abuse of official publicity as a resource for electoral campaigning. During the Mexican Presidential Elections in 2006 the five candidates aired 757,572 spots on radio and TV, while the Mexican President aired approximately 462,000 spots (2/3 of all such spots) publicising his image and governmental actions on social programs. Local governments also aired an unknown number of spots with similar characteristics, but they were not counted. It is important to highlight that this publicity was paid for with public resources. + +This is why this campaign is really important: it is not just about showing how much the government spends in this particular area, but it is also about measuring the impact of these continuous violations of this constitutional article e.g. on freedom of expression. + +[Read more on rules regulating government advertising in Mexico.](http://publicidadoficial.com.mx/como-se-regula-a-nivel-nacional) + +## What are the aims? + +Fundar approached this campaign with two concerns: + +* The first one is the concern about how the government spends its money in a more general context. +* The second concern is about the structural framework of democracy and the role of the media, as well as the relationship of the political parties and the media. + +There are two steps involved in tackling these concerns: + +* *Step 1:* Creating fair rules on how the governments can spend money on government advertising +* *Step 2:* Promoting information instead of propaganda in government advertising. + +A feasible goal for the project has been to gather data and evidence, which has not been available for government spending before and to provide it to civil society and the general public. In working with the data, there have been two aims: + +* Show what kind of data exists: get the numbers on government media spending. +* Denounce those parts of the government who refuse to provide the data for political reasons: show where the gaps are and where lack of transparency (opacidad) exist. + +## What kind of data and how to get it? + +The government does not release government advertising data for several reasons: + +* Deficient government accounting in Mexico due to structural problems in the bureaucracy. +* Trying to avoid political conflict, as in many states the media are not primarily a commercial, but a political actor, which can put a lot of pressure on the government. +* As some media is also owned by government officials, there is unwillingness to release the data which would reveal these connections and ownerships. + +A collective of organizations has worked on this topic since a while, but there have been no numbers to serve as evidence. For the Mexican civil society it is not very common to gather data and to try to provide it in some usable format. + +Learning from these organizations who were already involved in this discourse and had the political message, Fundar got the data as evidence. + +The data was obtained by Freedom of Information (FOI) requests, where some cases were brought even before the Supreme Court, which has to decide on the balance of public and private interest and whether the government had the right to protect information as a part of commercial negotiation. This aspect was contested by civil society, as public tenders should be public knowledge. In some cases a few FOI requests were done for the same kind of data in order to find whether there were any discrepancies. + +## Features + +The website of Publicidad Oficial offers: + +* A map and an index of transparency for all Mexican states (the red states are +where no information has been obtained) + + + + +* [Data on the government advertising of the federal +government](http://publicidadoficial.com.mx/gasto-federal) + + + + +
(Above) Index of access to spending information on government advertising per category: 11 of the Mexican states did not provide any information (0 stars). Only 2 Mexican states provide all the information - 5 stars.
+ +## Taxonomy of Data Problems - why was it difficult to work with the data? + +There were 3 main types of problems in working with the data: + +* Political Problems: it is often difficult to track how much money goes to a particular commercial entity. If ghost entities / businesses are created it might not be possible to track the spending to the actual company, as there is no actual proof for connecting the transaction back to them. +* Legal Problems: related to the legality of opening up government spending data when this data is related to a commercial negotiation. A lot of litigation currently is about the government disputing whether some tenders should be protected because of subject to a commercial contract secret. +* Technical / Operative Problems: from issues about transcribing PDF into spreadsheet format to diverging data formats, data quality and availability, +resulting from the fact that Mexican states have different transparency laws and data standards. +* Access to information problems: the discrepancies between the data have posed a problem of reliability of the information we access. + +Additionally, fragmentation of the data is also an issue, as data is provided in PDF format for every state, every 6 months, separately for each programme and it is not available to download in bulk format or accessible through API. Often the technical requirements for open data are too specialised and too complicated for CSOs with limited technical capacity and the non-technical staff of the government. + +## Relevance, Audiences and User-groups + +The website of Publicidad Oficial has become the official source of information on government advertising in Mexico - it is practically the only source. Even the presidential candidates and the media itself cites Fundar as the source of the data. + +During the latest presidential elections there has been a polemic about the candidate’s expenditure on the media where candidates have been discussing who is in fact spending more on the media campaign. One candidate has even accused another of being a project of the television media: “If the television makes presidents, you would be a president”. + +Users of the webpage and involved stakeholders in the campaign were journalists, researchers, other civil society organisations as well as political parties. + +## The Role of Technology + +In order to justify having more technical staff in the organisation, one should find more purposes beyond the experimental use of the technology. While there is a need to develop the technological skills of civil society, these should be complementary +to their advocacy skills. Technology is just one of the tools for advocacy. + +## Successes + +1) Having raised public awareness – government spending on the media is a public and national topic, especially in the time of the electoral campaigns where this kind of transparency is called for. + +2) The opportunity to be the most important and respected source for this information and having provoked the winning candidate to take this issue forward with the new government. + +3) Having broken the traditional barriers that Fundar faces when trying to reach out to more people in creating a website with its own audience. + +## Failures + +1) Despite the public awareness and national discussion, no legislation /public policy have been created to solve the problem. The question of how to convert the discussion into tangible results still remains. It does exist a lot of draft bill on the subject, but no one has been passed. See: for more details. + +2) Not having shown the data in a more creative way, related directly to the problems in the communication between the non-technical researchers and the technical staff. A more collaborative way of work should be explored. + +3) Not having been able to create and maintain a bigger coalition as a stable process. Involving other actors and sharing one’s political capital has been challenging, as other organisations might have felt they have been invited to “someone else’s party”. + +[*] [Basic Principles for the regulation of advertising, ADC.](http://www.censuraindirecta.org.ar/advf/documentos/4804c757c7e629.40711373.pdf) + +## Lessons learned + +We asked Justine Dupuys from the project what advice she would give to organisations embarking on similar missions in other countries. Here is what she suggested: + +1) It is really important to clearly define government advertising and what is acceptable use. + +2) In order to fulfil investigation’s objectives, we made a lot of FOI requests at federal and local levels. We suggest [constructing a very complete database with all the deadlines](https://docs.google.com/spreadsheet/ccc?key=0AvoV_cBqwo28dHMwUUQybms4Z3dxZ2hDMGQ2Tm5ucGc). It will help to follow up on these requests. + +3) Communication strategy for releasing the findings on governmental advertising is complicated. Traditional media are reluctant to inform on this issue because they are taking advantages of this situation. My suggestion is to think about creative way of communicate. It is the reason why we implemented a website and used social media [@publioficial](https://twitter.com/PubliOficial). + +*Our aim at OpenSpending is to bring together people working all over the world, be they techies or advocacy organisations, to find common patterns and solutions in how to work with and glean answers from publicly available data. Know someone we should contact? [Get in touch](http://lists.okfn.org/mailman/listinfo/openspending).* \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-10-03-spending-standard.md b/examples/openspending/content/blog/2012-10-03-spending-standard.md new file mode 100644 index 00000000..80e51c45 --- /dev/null +++ b/examples/openspending/content/blog/2012-10-03-spending-standard.md @@ -0,0 +1,24 @@ +--- +authors: +- lucy +redirect_from: /2012/10/spending-standard/ +title: Upcoming call - Spending Standard +--- + +**At OKFestival we had a fantastic session bringing together the IATI and spending community to talk about how we could make standards work for the users of government financial data. Now we want to keep the ball rolling. Next week, we will have a call to discuss how we can make a standard for transaction-level spending data happen...** + +## Details + +* *When?*: Thursday 11th October 7pm CEST, 6pm BST, 1pm EDT. [Other timezones](http://www.timeanddate.com/worldclock/fixedtime.html?msg=Spending+Standard+Call&iso=20121011T19&p1=37&ah=1) +* *Where?*: Via Skype +* *How do I join?*: Please add your name, agenda points and Skype I.D. to the [pad](http://wdmmg.okfnpad.org/community-2012-10-11), we'll add you pull you in. + + + +## What's up for discussion? + +Last week, we (well, Friedrich) published a straw-man draft of what a standard could look like for transaction level. + +The draft is here: + +We're really keen for your thoughts and also to know who we might be able to engage as early adopters in this so please do drop us a line via the [mailing-list](http://lists.okfn.org/mailman/listinfo/openspending). \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-10-20-openspending-lemonde.mdown b/examples/openspending/content/blog/2012-10-20-openspending-lemonde.mdown new file mode 100644 index 00000000..6d693a38 --- /dev/null +++ b/examples/openspending/content/blog/2012-10-20-openspending-lemonde.mdown @@ -0,0 +1,19 @@ +--- +authors: +- lucy +redirect_from: /2012/10/openspending-lemonde/ +title: OpenSpending visualisations featured in Le Monde +--- + +The OpenSpending platform experienced a huge peak in traffic earlier this week as a visualisation based on French data was featured in Le Monde. + +[The article](http://www.lemonde.fr/politique/article/2012/10/16/plf-des-avions-au-bouclier-fiscal-la-java-des-amendements_1776093_823448.html?xtmc=depenses&xtcr=52) "PLF : des avions au bouclier fiscal, la java des amendements", (PLF=Projet de loi de finances, the draft finance law) deals with suggested amendments to the draft finance law and which parties were demanding what amendments. + +The OpenSpending visualisation used in the article is intended to give a high-level representation of some of the main areas of government expenditure in France: + + + +Besides the OpenSpending visualisation, there are some simple but effective infographics on how many amendments were filed to the draft, and by whom. + +*Read the full article on [Le Monde](http://www.lemonde.fr/politique/article/2012/10/16/plf-des-avions-au-bouclier-fiscal-la-java-des-amendements_1776093_823448.html?xtmc=depenses&xtcr=52).* +*A slightly different view of the visualisation was also featured in Libération. Read the piece [here](http://www.liberation.fr/politiques/2012/10/18/budget-comment-depenser-371-milliards_854160).* diff --git a/examples/openspending/content/blog/2012-10-22-open-interests.md b/examples/openspending/content/blog/2012-10-22-open-interests.md new file mode 100644 index 00000000..4d321736 --- /dev/null +++ b/examples/openspending/content/blog/2012-10-22-open-interests.md @@ -0,0 +1,28 @@ +--- +authors: +- friedrich +redirect_from: /2012/10/open-interests/ +title: Open Interests Hackathon - November 24-25, London, England +--- + +As a journalist, to understand European Union institutions, policies and commitments, you have to look where the money goes and understand who affects the money flow in the EU. As the influence of Brussels lobbyists grows, it is increasingly important to draw the connections between lobbying, policy-making and funding. [The EU publishes information on its spending](http://ec.europa.eu/beneficiaries/fts/index_en.htm) and also maintains a [transparency register](http://europa.eu/transparency-register/). These, however, are difficult for journalists and citizens to use. + +With [OpenSpending](http://openspending.org/), we set out to use the power of technology to catalyze greater government transparency by providing new tools for media and citizens to more easily access government data in searchable, sortable and machine readable formats. + +Hackday-thumb-800x600-2721.jpeg + +One step on our road to achieving this goal is a "hackathon" where we aim to build tools that will help the media and citizens investigate the influence of Brussels lobbyists and where the money goes in the EU. [Open Interests Europe](http://okfnlabs.org/events/hackdays/lobbying.html), which will take place on November 24-25 at the Google Campus Cafe in London, brings together developers, designers, activists, journalists and other geeks for two days of collaboration, learning, fun, intense hacking and app building. + +Below are some of the projects we will be working on. + +## The Lobby Transparency Challenge + +Within any political process there are many interests wanting to be heard -- companies, trade unions, NGOs -- and Brussels is no exception. Corporate Europe Observatory, Friends of the Earth Europe and LobbyControl have begun to data-mine the lobby registers of the European Commission and of the European Parliament to find out who the lobbyists are, what they want, and how much they are investing. Participants will have the exclusive opportunity to work with this data before it is made public in their upcoming portal. + +## The Fish Subsidies Challenge + +Subsidies paid to owners of fishing vessels and others working in the fishing industry under the European Union's common fisheries policy amount to approximately 1 billion euros ($1.3 billion) a year. EU Transparency gathered detailed data relating to payments and recipients of fisheries subsidies in every EU member state from multiple sources, from European Commission databases to member state government databases and inter-governmental fishery organizations such as ICCAT. Participants will have the opportunity to build maps, visualizations and apps with this data. + +There will be prizes and interesting talks by jury members [Rufus Pollock](https://twitter.com/rufuspollock), co-founder and director of the [Open Knowledge Foundation](http://okfn.org/), and [Alastair Dant](https://twitter.com/ajdant), lead interactive technologist for the Guardian. The hackathon is organized by the [European Journalism Centre](http://www.ejc.nl/) and the Open Knowledge Foundation, and sponsored by [Knight-Mozilla OpenNews](http://www.mozillaopennews.org/). + +*You can find out more details on the [event webpage](http://okfnlabs.org/events/hackdays/lobbying.html). If you'd like to join us and help build these tools, please register [here](http://openinterests.eventbrite.com/).* \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-11-13-handbook-spring-day-1.md b/examples/openspending/content/blog/2012-11-13-handbook-spring-day-1.md new file mode 100644 index 00000000..5d765957 --- /dev/null +++ b/examples/openspending/content/blog/2012-11-13-handbook-spring-day-1.md @@ -0,0 +1,26 @@ +--- +authors: +- lucy +redirect_from: /2012/11/handbook-spring-day-1/ +title: Spending Data Handbook Sprint - Day 1 +--- +As you read this, we're writing a book about spending. It'll be finished by Thursday so you can read it over the weekend, as you sit down to do, (or not do) your armchair auditing of government accounts. As we get the opportunity to meet budget and data activists from all around the world, a set of common issues emerge and some novel and interesting approaches to dealing with them. The purpose of this week's work is to encapsulate these issues and ideas, tips and tricks into a format that will be useful to all manner of organisations working in this area. + +The topics are getting data, handling it and present it to various audiences and is aimed at answering the questions: "How can CSOs, open data activists and governments work together to make sense of spending information and to hold government to account?" + + + +Over the coming few days, we will publish our progress and ask you to help us to fill in the gaps. What we hope to have at the end is a living document, the beginnings of a toolkit which NGOs (and possibly public servants) can use to help liberate more data, collaborate with other organisations to do ever better analysis and policy-making and present their results in a way to produce meaningful responses from their citizens. + + + +## What's happened - Day 1 + +The philosophy of a book sprint is that the book is created from scratch, by the participants from the table of contents to the final gloss and polish, hence today was spent shaping the scope. Many post-it notes, scribbles and head scratches later, we have a preliminary table of contents. + + + +This includes workflows and working practices for NGOs, a list of demands which CSOs could put to governments in their country to get better access to types of data they need, tips and tricks which the techies use in their work which could be useful for NGOs and crucially - how to present your data to leverage input from the key audiences you are targeting. We believe that the current state of play has been unsuccessful in creating "armchair auditors" - it seems likely that many conversations will take place over the coming days as to whether anything could be done to change this, using technilogy or wit. + + + diff --git a/examples/openspending/content/blog/2012-11-15-handbook-draft.md b/examples/openspending/content/blog/2012-11-15-handbook-draft.md new file mode 100644 index 00000000..4ec76b98 --- /dev/null +++ b/examples/openspending/content/blog/2012-11-15-handbook-draft.md @@ -0,0 +1,35 @@ +--- +authors: +- friedrich +redirect_from: /2012/11/handbook-draft/ +title: Spending Data Handbook - Draft Version +--- + +After four days of intense writing, discussions and editing, it is now +almost midnight: our handbook sprint is over. What we have produced is +an introduction to the use of data for budget- and spending-focussed +advocacy. We've produced a high-level overview that covers a variety of +topics, from the context in which data can be used, to the acquisition, +processing and presentation of budget and spending data. + +Of course, four days means we have only been able to lay the groundwork: +we would still like to add more concrete tips and tricks, examples and +in-depth tutorials on a few technologies and tools that relate to +financial information. + +So, here it goes the first draft: + +Spending +Data Handbook + +Of course, this can merely be the start of a discussion. Please feel +free to contribute directly in the [editing +environment](http://okfn.booktype.pro/spending-data-handbook), or via our +[mailing list](http://lists.okfn.org/mailman/listinfo/openspending). + +We hope you enjoy the read! + + + + diff --git a/examples/openspending/content/blog/2012-11-26-Sarajevo-Workshop-Writeup.mdown b/examples/openspending/content/blog/2012-11-26-Sarajevo-Workshop-Writeup.mdown new file mode 100644 index 00000000..b796d974 --- /dev/null +++ b/examples/openspending/content/blog/2012-11-26-Sarajevo-Workshop-Writeup.mdown @@ -0,0 +1,50 @@ +--- +authors: +- lucy +redirect_from: /2012/11/Sarajevo-Workshop-Writeup/ +title: Day 1 OpenSpending CSO Workshop - Sarajevo +--- + +*A while back, we wrote about the [kickoff of our project to deliver the budget of Bosnia and Herzegovina to its citizens in a form they can understand](http://openspending.org/blog/2012/09/26/Balkan-Budgets.html). Last week in Sarajevo - we had the kickoff workshop, bringing together a group of techies and policy experts from the Balkans and Eastern Europe, the OpenSpending team and MySociety's Tony Bowden to see how, through and beyond visualisation, we could work together to make budgets in the Balkans and Eastern Europe more transparent and accountable.* + +# Day 1 - Inspiration and Open Data + +

+ +The OpenSpending team has spent a lot of time training journalists on how to use the both the OpenSpending platform and financial data in general, however this is the first time we've had the opportunity to train people whose aim was not necessarily to highlight scandal and sensation, but to systematically analyse and inform policy based on the available data. All of our participants had an additional aim besides improving policy: to answer the question "how do we display budget and spending information to citizens in a way that is engaging, meaningful and may even produce some action?" + +

+ +First up, an introduction to Open Data, to make sure everyone is on the same page. The aim of the day was inspiration to make data projects as powerful as possible, so to kick it off, Friedrich Lindenberg and Lisa Evans showed examples of data-driven financial projects which they felt had really made an impact in society: + + + +* **[The Farm Subsidies Project](http://farmsubsidies.org/)**: A collection of investigative journalists who pull together an enormous European Agricultural Subsidies Database and find people and companies using funds originally intended for small farmers. +* **[The UK's The MP's expenses scandal](http://en.wikipedia.org/wiki/United_Kingdom_parliamentary_expenses_scandal)**: After a huge name and shame campaign by a variety of major news outlets highlighting all manner of *innovative* uses of public money (from buying duck houses to claiming for fictitious second homes), a decision was made to proactively publish the expenses claimed by MP's when they happened. This has been so successful that MP's have really had to clean up their act, so successful in fact that the Guardian recently wrote that the story was getting boring, no-one was doing anything scandalous with it anymore. +* **[Free the files Campaign by ProPublica](http://www.propublica.org/series/free-the-files)** - tracking political ad filings from television stations in swing markets. Television stations are required to maintain a “political file” of political ads requests and contracts and ProPublica helps to make these searchable and easier to identify trends and culprits who abuse the system. + + + +After lunch, the participants were on stage to present their existing and proposed projects by way of further inspiration for the other groups in the room. Each group had taken a distinctly different approach to the topic of making decisions about public money in their country better. We heard from Expert Grup in Moldova on their proposed project to convert the Moldovan BOOST data into a format which could be understood by citizens, CRTA from Serbia on [PratiPare](http://www.pratipare.rs/) - a project to track the location and actual cost of a variety of projects in Serbia, ranging from schools to highways, Open Data Albania's use of Linked Data to connect spending to a variety of different other sources of Data. Lastly, [OneWorldSee](http://oneworldsee.org/) and [Centre for Public Interest Advocacy](http://cpi.ba/), Bosnia took to the stage, describing some of their past and up and coming projects, including CPI's 'Balkan Mythbusters'. We wait with anticipation. + +Next up - a great talk from My Society's Tony Bowden building on his experience with My Society's projects on how to build a useful and world-/game-changing project. Besides his key tips involving headaches illustrated above - he had the following tips for people who wanted their projects to change the world. + +
+ +Tony's Tips +
    +
  • Think about who would possibly pay for your services. If someone would pay for it, it's probably valuable (and you'll have a business model when the funding runs out). Think - 'What Would Apple Do?' ;). This is not to say that loss-making projects are not valuable, but bearing this in mind could help to think about sustaining your efforts.
    • +
  • Measure the actions completed your site, not numbers of visitors.
    • +
  • While we're on the topic of measuring people visiting your site. Is it possible that one of the most successful projects could be one in which almost no-one at all visits your site, however, the existence of said site stops bad people from doing bad things as they know people could check at any time? +
  • Make as much of the admin interface public as possible. People should never need your permission to come and help you out and it's surprising how many people will want to.
    • +
  • Get people with obsessive levels of interest on your side as early adopters of your site. The first people who start to use your site will most likely be people you already know. The first time you spot someone who you don't know, who arrives to use your tool - try and find out what made them find it - what made them want to take part. In the build up to the launch of Fix My Transport, My Society made friends with many train spotters who were happy to help out to improve the objects of their hobby.
    • +
  • Introduce league tables for people who help you out. People are instinctively competitive and like to see their name acknowledged . MySociety regularly use two league tables in parallel - one for people who have helped most in the last week, one for people who have helped most of all time. It's a nice acknowledgement for people to let others see how much they have contributed, and may in some circumstances, encourage people to participate more or for longer.
    • +
  • A note on crowdsourcing. A lot of people try to 'crowdsource' information. A lot of projects fail and are not kept up. There is a danger here that the people who put their efforts into providing this information will become disillusioned with the fact that nothing has been done with it. However, if you are planning on collecting information yourself as part of your research anyway, why not open it up to the public and see whether there is any way they can help you? The benefits of this are - you'll be putting the early data in anyway, making your site look popular and encouraging people to come and fill it in. If no-one comes, you don't lose anything - if they do - you save yourself lots of work!
    • +
+
+ +Finally - an introduction to the OpenSpending project for those not familiar with it. We show the ways in which the project has been used, go through a few of our mistakes which were made on the way to creating what is now a pretty stable platform - urging participants not to repeat them, as well as some of its success stories. + +We show that using OpenSpending doesn't mean you have to produce a cookie-cutter version of [Where Does My Money Go?](http://wheredoesmymoneygo.org/), and in fact, we'll get grumpy if you're not more ambitious than that. There's no licence (I'm aware of) to enforce this - but we want anyone who uses the [Assembly Kit](http://openspending.org/blog/2012/02/16/thekit.html) to build their own site to add something, however small, to make it better. We've done this recently with a project in Cameroon, soon to be launched, including a sub-national transparency index, per capita calculations and searchable data into the mix. This is going to be a theme for the next day, can't wait to see what people come up with. + + diff --git a/examples/openspending/content/blog/2012-11-27-Sarajevo-Workshop-Writeup-2.md b/examples/openspending/content/blog/2012-11-27-Sarajevo-Workshop-Writeup-2.md new file mode 100644 index 00000000..fd54c758 --- /dev/null +++ b/examples/openspending/content/blog/2012-11-27-Sarajevo-Workshop-Writeup-2.md @@ -0,0 +1,49 @@ +--- +authors: +- lucy +redirect_from: /2012/11/Sarajevo-Workshop-Writeup-2/ +title: Day 2 OpenSpending CSO Workshop - Sarajevo +--- + +This blogpost is the second in the series of the OpenSpending workshop - Sarajevo. Read the [first post](http://openspending.org/blog/2012/11/26/Sarajevo-Workshop-Writeup.html) on the OpenSpending blog. + +# Day 2 - Converting Data into Action & Finding Narratives in Data + +The people we are trying to target with this data are bombarded constantly from every side with data, numbers statistics - what they need is *narratives*, things that stick in their mind when they leave your website. This is the theme of day 2 - How to convert data into action: how to find the times at which people will be receptive to your message, and how to create a narrative they will remember. To get things rolling, we have a quick look at some of the other projects going on around the world, and play a game to see how much information people can extract from a visualisation in 30 seconds. + + + +Before splitting off into two streams for the day: *technical* and *policy*, we quickly show everyone together how easy it is to upload and model your data using OpenSpending. If anyone missed it, or couldn't follow along due to our lovely and mighty wifi connection, we've [documented this](http://cameroon.openspending.org/en/contribute.html) extensively as part of the World Bank Cameroon project. + +## Policy Stream + + + +
You can see the full-sized file on our Flickr Stream
+ +The policy stream highlight the areas they are interested in capturing in their projects, from international comparisons, to naming and shaming and then ask the key question: "Who cares?" - to work out who we should be targeting and via which medium. Secondly, we did a deeper dive into some of the problems experienced by the various projects, including how to get the data you require when the people giving it to you know you are competent enough to find scandal in it. + + + +### Targeting citizens with your project + +**PERSONAL BELIEF ALERT!** It's a blunt statement, but I'm going to put it out there anyway: I believe citizens don't care about budgets. At least, they don't care about budgets at large, the majority of the time. When numbers are presented to them as ten figure numbers, it is very tricky for them to identify with as they don't relate to things of a size of anything familiar: A household budget, the turnover of their business. + +What citizens might sometimes care about is the overall funding priorities of the government as this affects the services they require from government - is my government spending more on defence than it is on hospitals or schools? Journalists have a nose for this, they can feel when people are likely to be receptive to messages like these, and we at OpenSpending often see a traffic spike around the budget times of various countries as they embed our visualisations in their sites and as policy makers themselves use the visualisations to explain freshly drafted budgets to other policymakers (no jokes). + +So, the challenge is: beyond key points in the budget year, how do we keep people interested in how their government spends their tax money? What they might care about is particular things, often **services**, such as hospitals or schools, which they might want improving. We, as policy people and techies, who are prepared to crunch the numbers, deliver them in a way which connects them to the services people care about? + +### Targeting the experts + +There are however, plenty of people who do care about the real numbers - NGOs, Researchers, Internet Activists and Policy Wonks - is there a way that some of our projects can remove the headaches for working with data for these groups? At OpenSpending - we've already seen a demand for the ability to search transactions for [supplier information](http://openspending.org/blog/2012/02/24/how-spending-stories-fact-checks-big-brother-the-wiretappers-ball.html), and via the mailing list, we're collecting a list of questions which other people would like to be able to answer. Could organisations such as the ones at the workshop, actually become a better source for this type of data than governments themselves if they can combine it, present it and query it better? + +## Technical Stream + +The technical stream split off for an action-packed day. They cover an introduction to [DataWrapper](http://datawrapper.de/) for making simple charts and web visualisations, [Kartograph](http://kartograph.org/) for making elegant maps, Scraping using [ScraperWiki](http://scraperwiki.com/), using Optical Character Recognition to get data out of PDFs and cleaning data using [Google Refine](http://code.google.com/p/google-refine/). All before 5:30 :) + +## Project Proposals + +The wrapup activity for Day 2 was a Dream Project proposal. If money and data were no object, what would the participants build? A variety of projects, from a Fix My Street-esque project for Bosnia, could be used to produce some alternative performance statistics for various project to rethinking how budget laws were made, so that they had to be submitted as small pieces of code. + +While the suggestion from the group from CRTA and Tony Bowden to build on their project tracking site by equipping kids with cameras to take pictures of broken parts of playparks was, I believe, intended largely jokingly, I can't help wondering whether encouraging kids as part of their school projects to take part in these projects might not be a bad idea. Other large management consultancy companies use this technique and besides just collecting data, you are teaching the kids to be active citizens. I shall continue to ponder... + diff --git a/examples/openspending/content/blog/2012-11-28-Sarajevo-Workshop-Writeup-3.md b/examples/openspending/content/blog/2012-11-28-Sarajevo-Workshop-Writeup-3.md new file mode 100644 index 00000000..997e83a9 --- /dev/null +++ b/examples/openspending/content/blog/2012-11-28-Sarajevo-Workshop-Writeup-3.md @@ -0,0 +1,52 @@ +--- +authors: +- lucy +redirect_from: /2012/11/Sarajevo-Workshop-Writeup-3/ +title: Day 3 OpenSpending CSO Workshop - Sarajevo +--- + +This blogpost is the last in the series of the OpenSpending workshop - Sarajevo. Read the [first post](http://openspending.org/blog/2012/11/26/Sarajevo-Workshop-Writeup.html) and the [second post](http://openspending.org/blog/2012/11/26/Sarajevo-Workshop-Writeup-2.html) on the OpenSpending blog. + +# Day 3 - Getting your message out + +As a CSO, once you have done all of this work, how can you make sure it is used? The focus of Day 3 was getting your message out there. + +## Policy Stream + +The Policy Stream actually subdivided even further into Data Analysis and Public Relations for the day. + +### Data Analysis + +Some of the participants were interested in more in-depth data analysis, so using the Moldovan BOOST data - we tried to generate some potential leads. Some interesting possible avenues for future exploration emerged including a surprising lack of money going to hospitals in the Moldovan capital - more investigation required to work out whether this is actually the case, or just gaps in the data, but an interesting lead nonetheless for the Moldovan group, who had already identified healthcare as an area they wanted to look into further. + +### Communicating with the Media and Outreach + +A tricky and frustrating subject for Think Tanks and CSOs who work with this type of information is how to get their outputs used. Again, we took a combination of the wacky dream approach and tried and tested experience from the participants. How could we revolutionise the press-release so that people actually used it? Would giving them easy ways to drop your visualisations into their articles increase uptake? Inspired by the [Obama for America Campaign, where targeted messages were used to great effect](http://projects.propublica.org/emails/) to target different parts of the audience, was there a good (and not too labour intensive) way to tailor your message to target the different types of media organisations? Loads of great ideas, which I tried to scribble down in our mindmap. + + + + +
You can see the full-sized file on our Flickr Stream
+ +## Technical Stream + +The technical stream got down to work with putting into action the skills they had learned on the previous day and how to build a custom tax calculator for their country. There's a blog post with more instructions on this one cooking, we'll be in touch when we have it ready! + +# Stay in touch + +We'd love to stay in touch and for other organisations to join the discussion on how we can take these projects all to the next level and hopefully collaborate even more internationally. + +The two mailing lists we regularly use for this type of communication are: + +* "Policy stuff": +* Technical discussion: + +We hope to see you there soon! + +# Our Slides: + +Introduction to the workshop: + + + +*We'll add more as we get them, still waiting for them to come trickling in!*. diff --git a/examples/openspending/content/blog/2012-11-30-Bosnia-Budget-Classification.mdown b/examples/openspending/content/blog/2012-11-30-Bosnia-Budget-Classification.mdown new file mode 100644 index 00000000..a6bb1af5 --- /dev/null +++ b/examples/openspending/content/blog/2012-11-30-Bosnia-Budget-Classification.mdown @@ -0,0 +1,43 @@ +--- +authors: +- lucy +redirect_from: /2012/11/Bosnia-Budget-Classification/ +title: Bosnian Budgets - grouping data by categories people care about +--- + +Last week, I sat watching the team of CPI Bosnia mapping the Bosnian budget into functional classifications. We're working on making the budget accessible to the citizens of Bosnia, making clear visually things such as the division of funds between the cantons and municipalities of the country. + +*Functional classifications*, for those of you who don't regularly 'sail the wide accounting seas' tell you things like what general area of spending we area talking about "health", "education", "defence", which is often more interesting from the perspective of the citixen user than e.g. which ministry group it was spent by. + + + +The evening would probably be a little more fun if someone inside government had done it for them, so we could go out and have beer, but nevertheless, it's important to get this right. No idea what we are talking about or why you should care? Read on: + +## Why Functional Classifications? + +Simply speaking, many users of data want to know *what* government spent money on, rather than who spent it, who received it. People (I'm talking about the general public here) generally care about services - not bank transfers. + +You don't have to make these classifications up from scratch, there are internationally recognised systems of these. For example, the stylishly named [Classifications of Functions of Government (COFOG, for short)](http://unstats.un.org/unsd/cr/registry/regcst.asp?Cl=4) - is how the government already publishes its data in the UK. This, with a few amendments (see De-jargonising COFOG) - was the system used to make the budget understandable in Where Does My Money Go? + +For other projects which we've done e.g. [Cameroon.OpenSpending.org](http://cameroon.openspending.org/en/) we've used a COFOG-esque mapping. Why 'esque'? Firstly, the government didn't publish their data classified like this, so we had to group it ourself. Secondly, we were aiming here for a functional classification which worked when you visualised it, if we'd used COFOG exactly, we'd have ended up with a huge bubble for general public services which would have made all the others really small, so you wouldn't be able to see the difference in size. So we modified the set of top-level items to make it easier to see smaller distinctions. + +For the first version of the Bosnia project, we've got functional classifications for the top level, then bodies which spent the money for the second level of our visualisation. See what they've done in this [Google Doc](https://docs.google.com/open?id=1tyfmH9EqKz_3VucDQWGmKIPpcSPYg6iCcDzwH1wwbdNJvZqoUnTnYRcmlNhV). + +### International Comparisons + +While we're always warning people about making comparisons between countries (data not being collected in the right way etc lalala), these classifications using COFOG are quite often used to make international comparisons. [OECD do it regularly](http://stats.oecd.org/Index.aspx?QueryId=30428), so it's probably one of the less-evil ways to do it, in case you're interested in that type of thing. + +## De-jargonising COFOG + +Let's face it, the terminology used by the government is often not the most appealing, or illustrative from the point of view of the user. Hence, for the Where Does My Money Go Project, we specifically de-jargonised it, and translated the terms into friendly forms that we felt were more accessible to the average user. For example: 'Executive and legislative organs, financial and fiscal affairs, external affairs' became 'Top level government'. You can take a look at how we mapped them on to one-another [in this Google Doc](https://docs.google.com/spreadsheet/ccc?key=0Ah8UkI7xG7eWdFFTSjlkeFRoOEFLbC1PTjRRcWphOFE#gid=0). + +### How to map your budget into COFOG classifications: + +Basically, if your government doesn't do this for you - you'll always have to use your best judgements, someone may have made a different call, and may well disagree with the way you've done what you've done. But as long as you document your practices, anyone will be able to pick up anything they don't agree with and produce a different model. So as I'm sat here, I'm listening to people bandying about terms and trying to decide which ones are most relevant. + +1. Make a codesheet and align your functional groups to the things you want to go under that umbrella term. +2. Do a [dataset cross](https://docs.google.com/document/d/1bD3KztcPdc3Ffe5_xlVl--N2wBVydUtpUDJpq2d6sK8/edit#heading=h.d1ub48are7ej) using Google Refine or use HLOOKUP in Excel. Dataset merging will allow you to match information from different data sources or spreadsheets, without merging them, so the original data remains available. + +## Use this method other places. + +By the way: this methodology is exactly the same as you would need to syncronise geographic information. E.g. if you've got names in one format e.g. full names and you need them in another format, e.g. [ISO-3166](country codes) - you can easily use a code sheet and the same dataset cross techniques. diff --git a/examples/openspending/content/blog/2012-12-18-Jobs-OS.mdown b/examples/openspending/content/blog/2012-12-18-Jobs-OS.mdown new file mode 100644 index 00000000..4e6aa890 --- /dev/null +++ b/examples/openspending/content/blog/2012-12-18-Jobs-OS.mdown @@ -0,0 +1,32 @@ +--- +authors: +- lucy +redirect_from: /2012/12/Jobs-OS/ +title: We're hiring! +--- + +*Hacker? Passionate about finding stories in the money? Care about which companies get contracts with government? OpenSpending is seeking a lead developer to guide it through its next phase of development and it might be just the job for you!* + +OpenSpending is about mapping the money. We want to make government finances accessible to advocates, journalists and citizens. Our goal is to collect budget and spending information from across the world and to present it in a form that promotes understanding, analysis and participation. Some of the questions we ask are: + +* How much is government spending on health? Is expenditure growing or shrinking? How does this translate into results? +* What are the proportions of different government programmes? What is spending on prisons compared to schools? How much is Ghana spending on education compared to Nigeria? +* How much taxes do I pay into which area of government? +* Our day-to-day work has many facets: working on the core platform, journalistic projects as part of our Knight News Challenge 2011 winning entry “Spending Stories” and working with organizations and civic activists world-wide to set up local budget transparency projects. + + + +## About the role +We’re looking for a lead developer and evangelist to maintain and further develop OpenSpending and the Spending Stories project. + +Some things we look for: + +* Strong interest in open government and transparency +* Fluent in JavaScript, Python and HTML5/CSS (include links to any sites you built and code repositories, e.g. GitHub, BitBucket) +* Experience with data warehousing, ETL, data processing and management techniques a big plus. +* An appreciation of design and beautiful things +* Readiness to do travelling, some writing, public speaking and to promote OKFN projects +* This person will work closely with the Head of Knowledge to ensure that these activities integrate with the rest of the OKFN’s activities. + +## How to apply +If you’d like to apply, please email jobs@okfn.org with the subject line “OpenSpending Tech Lead” before 7th Jan. Please include links to some demos of work from your portfolio, and your CV. \ No newline at end of file diff --git a/examples/openspending/content/blog/2012-12-24-Roundup.mdown b/examples/openspending/content/blog/2012-12-24-Roundup.mdown new file mode 100644 index 00000000..26fb1dab --- /dev/null +++ b/examples/openspending/content/blog/2012-12-24-Roundup.mdown @@ -0,0 +1,76 @@ +--- +authors: +- lucy +redirect_from: /2012/12/Roundup/ +title: OpenSpending A Year in Review +--- + +Just before I turn on my autoresponder and submerge my computer into the blackness of holiday in a concrete box to which I have no access, I thought I'd just quickly wrap up a few of the highlights from this amazing year. It has been very intense - but we've covered a lot of ground, here's a few highlights... + +## January + +The year kicked off and we launched the Spending Data Working group. This fantastic group of people include some of the world experts and techies who are passionate about linking up the money flows across the world. They meet and natter via the [OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending) - drop them a line and join the conversation! + +## February + + + +We were also approached by Privacy International, to ask whether they could use [OpenSpending's global search to find out which governments were purchasing surveillance equipment to spy on their citizens](http://openspending.org/blog/2012/02/24/how-spending-stories-fact-checks-big-brother-the-wiretappers-ball.html). My favourite spending story so far - we had no idea people would use OpenSpending like that! + +We also headed out to Bosnia for the first time, where we met the Centre for Public Interest Advocacy - with whom we have now launched a project to build a version of Where Does My Money Go for Bosnia...Luckily - we'd just packaged up the [Where Does My Money Go Assembly kit](http://openspending.org/blog/2012/02/16/thekit.html), to make it a lot easier for people to build their own versions of the site... + +## March - May + + + +We put on our scout hats and set out to find some of the most interesting uses of Technology to promote Fiscal Transparency and showcased them in the [GIFT report](http://openspending.org/resources/gift/index.html). We're still looking for these, so if there are any you'd like to point us to [let us know](http://lists.okfn.org/mailman/listinfo/openspending). + +## April + +Off to Perugia for an epic data-journalism training session with the European Journalism Centre... At Perugia the [Data Journalism Handbook](http://datajournalismhandbook.org) launched. Also in April, we built the widgets that allowed anyone to embed OpenSpending's visualisations in a website/blogpost (like this - Nigeria's proposed budget for 2013!:) + + + +We promise we'll write up the widgets soon, we really should tell people about them! + +## June + +We finally got a BETA version of the IATI dataset loaded into OpenSpending. There are still a lot of gaps to fill. But it's up! Interested in helping to fill the gaps - the guys on the [Open Development List](http://lists.okfn.org/mailman/listinfo/open-development) are your friends. + +## July & August + + +*Image credits: [quapan](http://www.flickr.com/photos/hinkelstone/) on Flickr* + +Lucy kicked off the [Athens to Berlin series](http://openspending.org/blog/2012/07/05/OSI.html) travelling Europe to get a better understanding of the problems faced by organisations working in the field of government financial transparency all around Europe... Then - no rest for the wicked - she and [Laura Newman headed out to India](http://in.okfn.org/2012/09/18/okfn-india-trip-the-roundup/) to continue the quest. + +## September + + + +OKFestival! And we had a fantastic panel - [the Money and the Many](http://openspending.org/blog/2012/09/24/OKFest-Followup.html) bringing together members of the working group on open spending data from 3 continents to share their experiences on barriers and challenges in engaging citizens en masse to care about the issues presented in budgets and spending. + +We also worked with data.gov.uk to help build [an oversight tool](http://openspending.org/blog/2012/09/13/uk25k-reporting.html) to see which departments were complying with their transparency obligations... handy... + +## October + + +We got our act together and did something we'd been meaning to for ages. Drawing up a straw-man for a [standard for transaction-level spending data](http://openspending.org/resources/standard/index.html), and the discussions began. We'll continue these discussions in the New Year. Interested in adding your thoughts? [Here's your channel](http://lists.okfn.org/mailman/listinfo/openspending). + +A bonus for us was that both Libération and Le Monde used OpenSpending widgets to illustrate their articles on debates around the newly announced PLF. Both slightly different takes on the data and had sliced it in different ways - great to see... + + + +## November + + + +Ouch - November was packed. We kicked off at MozFest with the [School of Data Expeditions](http://schoolofdata.org/2012/11/14/data-expeditions-at-mozfest/), where our lovely new addition to the team, Lisa Evans lead the mini army of tax evasion-spotters. From there ran headlong straight into a 4 day Spending Data Handbook Sprint and then headed out to Bosnia for the [kickoff workshop of our Bosnia project](http://openspending.org/blog/2012/11/26/Sarajevo-Workshop-Writeup.html). + +For Friedrich, who can never get enough of back to back travelling, it didn't end there. From Bosnia he headed straight back to London for the [Open Interests Hackathon](http://openspending.org/blog/2012/10/22/open-interests.html), while I headed out to New York to attend the second TABridge session - bringing together techies and transparency organisations from around the world and locking them into an enclosed space until they solve the transparency needs of the world using technology ;) Lots of great people met, *loads* of hatchling projects... + +## December + +And so it was December already. We're looking forward to welcoming on board new team members in January. With a bigger team to juggle all of the exciting projects coming up in the new year, we're really looking forward to seeing the results next year... + +**See you in January! And if you get a moment in between munching on mince pies and fancy doing some armchair auditing, you know where we are ;)** \ No newline at end of file diff --git a/examples/openspending/content/blog/2013-01-07-Welcome.mdown b/examples/openspending/content/blog/2013-01-07-Welcome.mdown new file mode 100644 index 00000000..4353d567 --- /dev/null +++ b/examples/openspending/content/blog/2013-01-07-Welcome.mdown @@ -0,0 +1,8 @@ +--- +author: $authornamehere +redirect_from: /2013/01/Welcome/ +title: Welcome Anders +--- + +Welcome, Anders! + diff --git a/examples/openspending/content/blog/2013-01-08-open-spending-calendar.md b/examples/openspending/content/blog/2013-01-08-open-spending-calendar.md new file mode 100644 index 00000000..6c044278 --- /dev/null +++ b/examples/openspending/content/blog/2013-01-08-open-spending-calendar.md @@ -0,0 +1,27 @@ +--- +authors: +- lisa +redirect_from: /2013/01/open-spending-calendar/ +title: An Open Call for Ideas - How Would You Use OpenSpending's Data Calendar? +--- +With the new year upon us, it's the time of year when you might be thinking about the calendars you use and wondering how they could work better for you. + +Just over a month ago, the OpenSpending team floated the idea of an open spending calendar + +The idea was to monitor some key open spending data sets -- the ones that require work to get the story and that show where government money is going. Journalists can select which data sets they're interested in, and the calendar will alert them when the data is due be released. When it's released, we'll suggest a host of ways to get a story in a second email with a link to where you can download the data. The suggestions for getting a story will be very specific to the data set, but examples are: + +
    +
  • Links to people who are experts on this data set
    • +
  • Related data sets and replies to freedom-of-information requests and suggested ways to combine them
    • +
  • Clean up the data if it's not already in a usable format
    • +
  • Step-by-step guides for how to interpret the data from our team of statisticians and software developers working on our sister project, the School of Data.
    • +
  • Links to previous stories on the same topic that were a success
    • +
+ +Now what we'd really like to know is: Would you sign up for this calendar, and if so, how would you like it to work? + +The World Bank offer an open data calendar. It mainly shows open data conferences, but also includes announcements and offers users the ability to add their own events and data releases. + +We are very keen to hear what you think, and would appreciate if you could take a little time to fill in the form below so we can gauge demand and cater the calendar to your exact needs. + + diff --git a/examples/openspending/content/blog/2013-01-09-privacy.md b/examples/openspending/content/blog/2013-01-09-privacy.md new file mode 100644 index 00000000..bf8055e7 --- /dev/null +++ b/examples/openspending/content/blog/2013-01-09-privacy.md @@ -0,0 +1,20 @@ +--- +authors: +- lucy +redirect_from: /2013/01/privacy/ +title: OpenSpending community member identifies unredacted private transactions in UK local public spending data +--- + +A freelance data specialist from our community recently got in touch to let us know that he had used OpenSpending to identify a number of privacy breaches in an individual dataset presented in OpenSpending. + +This was due to inconsistent redaction of sensitive data by the local authority. Whilst the majority of these payments were to organisations (hence probably not highly sensitive), there were also a few unredacted payments to individuals. The person who uploaded the data immediately notified their local council, who in turn referred this to their audit committee. + +As we take privacy very seriously, as a precaution we have temporarily taken down the UK Local Council £500 spending data, which also featured data from the council in question. + +This incident highlights the importance of proper procedures to ensure that data from public sector bodies is properly redacted before being published. The UK government produces a [guideline document for data publishers](http://data.gov.uk/blog/local-spending-data-guidance), which ensures that issues like this are prevented and hence very rare. + +Every day people from around the world make use of the database of more than 13 million transactions provided by OpenSpending. The information promotes transparency and helps citizens to hold governments accountable. UK open spending data is some of the best in the world and has already allowed people to get understanding and insight into government spending at a level never before possible. + +In this case we’re glad that a member of our community was able to flag up private transactions that should not have been published - leading to these swiftly being taken down. We hope this serves as a reminder for public bodies to thoroughly scrutinise transactional data before it is published. + +*The OpenSpending team can be contacted on info@openspending.org for any further questions on this matter. \ No newline at end of file diff --git a/examples/openspending/content/blog/2013-01-22-How-to-decrypt-a-budget.md b/examples/openspending/content/blog/2013-01-22-How-to-decrypt-a-budget.md new file mode 100644 index 00000000..a20e49cc --- /dev/null +++ b/examples/openspending/content/blog/2013-01-22-How-to-decrypt-a-budget.md @@ -0,0 +1,15 @@ +--- +authors: +- lisa +redirect_from: /2013/01/How-to-decrypt-a-budget/ +title: How to decrypt a budget? +--- +If you’ve ever read, or even better tried to explain a budget you’ll be completely aware that budgets are jargon fests. + +This would be fine if the budget was just for people who know the lingo, but in the case of a government's budget, well, that’s really something everyone who votes and pays taxes should have access to. + +So how do we make budgets accessible to the public? Well there are lots of examples of [journalists](http://www.fsteurope.com/news/is-cash-becoming-extinct/), [non governmental organisations](http://twaweza.org/uploads/flash/budget-visualization-kenya-000/Kenya.html#/home/split=Purpose&spending=Actual&viewType=Bubbles&year=2002-03) and [governments](http://www.flickr.com/photos/hmtreasury/sets/72157632177938360/) doing this. One of the masters of the use of innovative budget glossary is the European Union. For instance: What headline would you find appropriate for a programme, which covers €44 bn in direct farm subsidies and a mere € 0.2 bn for [conservation](http://ec.europa.eu/environment/life/) annually. Well, “[Preservation and Management of Natural Resources](http://ec.europa.eu/budget/financialreport/expenditure/naturalresources/index_en.html)” might not be the name you would find the most fitting. + +If you want to get started on a budget that is still carrying cryptic headers you might want to consult the extensive World Bank [budget glossary](https://docs.google.com/file/d/0B1gQoR-EKl_xdHJhRDlTTkc5WDA/edit). However, merely knowing what the terms mean often doesn’t help understand the bigger picture of how the budget works. For that you have to understand the terminology in context. The World Bank have this covered too with this [detailed description of the budgeting process](https://docs.google.com/file/d/0B1gQoR-EKl_xNUVYb1d2RTdjMFk/edit). This is a good starting point for working with budgets. + +If you find better or simpler ways to explain and work with budgets and want to let the community of open spending and budgeting enthusiasts know, then please start a discussion thread on the [open spending mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2013-01-24-obi-post.mdown b/examples/openspending/content/blog/2013-01-24-obi-post.mdown new file mode 100644 index 00000000..d8e0f754 --- /dev/null +++ b/examples/openspending/content/blog/2013-01-24-obi-post.mdown @@ -0,0 +1,23 @@ +--- +author: $authornamehere +redirect_from: /2013/01/obi-post/ +title: The Open Budget Survey is out +--- + +Yesterday the [International Budget Partnership] (http://internationalbudget.org/) published their [2012 Open Budget Survey] (http://survey.internationalbudget.org/#rankings) evaluating fiscal transparency across 100 countries. Based on the survey, IBP calculates the Open Budget Index, as a measure for comparing budget transparency across the world. The survey is based on the performance of countries across 125 questions on everything from the independence of state auditors to the transparency of each and every step of the budgeting process. + +This year's Open Budget Survey features a data explorer developed in coorperation with OKF. Here you're able to compare how the countries preform in the overall survey or on specific questions compared to their earlier scores back to 2006 in a [timeline] (http://survey.internationalbudget.org/#timeline). + + + +For each country you're also able to review [each question and score] (http://survey.internationalbudget.org/#profile/) in detail. So whether you're a researcher, journalist or simply immensely curious the survey should offer plenty of opportunities to dig deep into governmental budgetary transparency. + +## Join our Community Call on assessing fiscal transprency standards! +At OpenSpending we wanted to follow up on the Open Budget Survey by organizing a Community Call on Wednesday, January 30th 1900 CEST / 1300 EST. + +Topic: What questions to ask when assessing fiscal transparency standards of governments? + +To join: Simply add your name, SkypeID and agenda items [here] (http://wdmmg.okfnpad.org/16) + +If you're wish to follow the discussions on budget transparency and spending standards, don't forget to join our [mailing-list](http://lists.okfn.org/mailman/listinfo/openspending). + diff --git a/examples/openspending/content/blog/2013-01-24-open-budget-survey.mdown b/examples/openspending/content/blog/2013-01-24-open-budget-survey.mdown new file mode 100644 index 00000000..cbc0a449 --- /dev/null +++ b/examples/openspending/content/blog/2013-01-24-open-budget-survey.mdown @@ -0,0 +1,24 @@ +--- +authors: +- anders +redirect_from: /2013/01/open-budget-survey/ +title: The Open Budget Survey is out +--- + +Yesterday the [International Budget Partnership](http://internationalbudget.org/) published their [2012 Open Budget Survey](http://survey.internationalbudget.org/#rankings) assessing fiscal transparency across 100 countries. The survey is the most comprehensive of its kind and evaluates countries across 125 questions on everything from the independence of state auditors to the transparency of every step of the budgeting process. + +This year's Open Budget Survey features a [data explorer](http://survey.internationalbudget.org/) developed in collaboration with OKF. Here you are able to compare the performance of countries in the overall score or on any of the specific questions. IBP has conducted the survey every second year since 2006, and therefore it's possible to check if countries have delivered any progress in terns of ranking or score in a neat [timeline](http://survey.internationalbudget.org/#timeline). + + + +The survey also enables you to easily examine a specific country in detail as it provides [data on the assessment of each question of the survey](http://survey.internationalbudget.org/#profile/). So whether you're a researcher, journalist or simply immensely curious the survey should offer plenty of opportunities to dig deep into governmental budgetary transparency. + +# Join our Community Call on fiscal transprency standards! +At OpenSpending we want to take the opportunity to follow up and discuss the Open Budget Survey and therefore organize a Community Call on Wednesday, January 30th 1900 CEST / 1300 EST. + +Topic: What questions to ask when assessing fiscal transparency standards of governments? + +To join: Simply add your name, SkypeID and agenda items [here](http://wdmmg.okfnpad.org/16) + +If you wish to follow the discussions on budget transparency and spending standards, don't forget to join our [mailing-list](http://lists.okfn.org/mailman/listinfo/openspending). + diff --git a/examples/openspending/content/blog/2013-01-29-worldbank-guest-post.mdown b/examples/openspending/content/blog/2013-01-29-worldbank-guest-post.mdown new file mode 100644 index 00000000..3c92a673 --- /dev/null +++ b/examples/openspending/content/blog/2013-01-29-worldbank-guest-post.mdown @@ -0,0 +1,110 @@ +--- +authors: +- samuel +redirect_from: /2013/01/worldbank-guest-post/ +title: Can Open Data and Mobile Technology Eliminate the Need for a "Feedback" Loop? - Reimagining the new World Bank Open Finances app +--- + +This is a guest post by [Samuel Lee](https://twitter.com/OpenNotion) with contributions from [Francesco Ciriaci](https://twitter.com/fciriaci) and [Julia Bezgacheva](https://twitter.com/ulkins). + +Efforts to make spending information more relevant and actionable often suffer from low levels of financial literacy, complexity, and just the unfavorable common perception that financial data is boring and hard to understand. At some level, "money" is the universal common denominator. However at scale and without context, it becomes harder to conceptualize and may even become entirely abstract. The layer of "development finance" adds yet more complexity. In the Open Spending space, many of us have tried to bridge this gap on the path of development transparency towards citizen empowerment through open data, data visualization, social media, and mobile technology. + +As an embodiment of this same spirit, the [World Bank Open Finances](https://finances.worldbank.org/) team recently released a new mobile application [Android](https://play.google.com/store/apps/details?id=com.worldbank.finances&hl=en), [iOS](https://itunes.apple.com/us/app/world-bank-finances/id465555488?mt=8), [web](http://financesapp.worldbank.org/). While not ground-breaking, the thought process behind designing and the experience of building the app helps illustrate the challenges of working with financial data in an international development context and sheds light on what might be explored to take the next step. While [social media and mobile technology innovations](http://www.guardian.co.uk/global-development/poverty-matters/2013/jan/04/saving-world-social-media-development-digital) hold much promise and may one day make good on its promise to "save the world," we recognize there is much more that needs to be done to begin realizing that potential. We invite you to join the discussion and share your ideas. +
+## An Iterative Approach to Existing Challenges
+Dispelling the notion that financial or spending information is boring or difficult to understand relative to other data was an important mental block to cast off. As Dino Citraro of Periscopic puts it, "There is no such thing as 'good data,' there is only good context." Good context is definitely a key to making financial data more relevant and actionable. +So what does better context in a mobile format look like? Here's a breakdown of our latest attempt. + +Location + + + + + + +
+ + + + + +
+
+When someone looks at a photograph, often the first thing they do is to look for themselves. Given this dynamic, a pinchable map interface that gives users the chance to intuitively locate themselves in the larger "picture" made sense. Leveraging GPS technology and mashing up financial data with other data at the World Bank (projects, mapping, and procurement), the "Nearby" function helps users explore the question "What is the World Bank doing around me?" + +Language + + + + + +
+ + + +
+
+Have you ever tried to converse with someone who doesn't speak your language? The first World Bank Finances mobile app was only in English, and we quickly felt its limitations. Presenting information in multiple languages was a high priority for this release, which we believe will increase the probability of generating interest, spurring conversation, and moving towards regional impact through open financial data. For this version, the app is available in seven languages- Arabic, Chinese, English, French, Portuguese, Russian, and Spanish. (Please let us know what additional languages we should offer for future releases!) + +Share and Respond + + + + + + +
+ + + + + +
+
+While the app is not designed for more robust manipulation of data like our [web platform](https://finances.worldbank.org/), it allows interaction with financial data in a number of ways. The app offers the ability to share content through social media networks (Facebook and Twitter), e-mail, and SMS. Users also have the option to point out data accuracy issues and report allegations of fraud and corruption. + +Demand-Driven Approach (Contracts and Procurement Data)
+
+From the Development Data Challenge, June 2012
+ + + + + + +
+ + + +
+
+Often open data initiatives start as supply-driven efforts and sometimes continue as such. To add real value you need to answer real demand. We were fortunate to connect with like-minded partners to tap into creative views and perspectives. Through a collaborative "ideation" event ([Development Data Challenge](http://developmentdatachallenge.org/), a joint initiative of Publish What You Fund, Open Knowledge Foundation, the Guardian, and other partners) and a Random Hacks of Kindness hackathon, the demand for more open procurement data was [captured and responded to](http://blogs.worldbank.org/opendata/the-power-of-open-crowd-sourced-ideas-crowd-powered-solutions). An early prototype created by community developers was a critical proof of concept for parts of the World Bank Open Finances app and resulted in expansion of the Major Contracts dataset from [four months to seven years](https://finances.worldbank.org/Procurement/Major-Contract-Awards-FY2007-FY2013-Beta-version/kdui-wcs3). We are committed to this demand-driven approach and plan to explore more creative methods of documenting and responding to demand in the coming months. (Please let us know if you'd like to [get involved](http://blogs.worldbank.org/opendata/contact)!) +
+##Remaining Challenges
+Adding meangingful context remains a critical challenge and is perhaps a never-ending quest. Specifically, we feel these are the next areas that need to be addressed and welcome your contribution and collaboration in forging the path forward: + +1. Identifying Additional High Value Information
+Much like procurement data addressed a need and painted a more complete picture, we are keen to add more high value data towards a more accurate representation of the world. Some of this can also be accomplished through more regular data updates and markers of what has been added since a user's last visit. What other information would you include or mash-up? How might we better leverage the next [Development Data Challenge](http://developmentdatachallenge.org/) or upcoming ["Big Data" dive](http://europeandcis.undp.org/blog/2013/01/11/can-big-data-help-deliver-better-operational-results/)? + +2. Improved Information Delivery
+Multiple languages were the first step. Better visualizations are also another area ripe for exploration. As a community, we are long overdue to move beyond pie charts, bar graphs, and tree maps. It is easy to rely on traditional reporting models in a new package, arguably which is not "good context." What can we draw from the [approach of and grafitti artists](http://infosurhoy.com/cocoon/saii/xhtml/en_GB/features/saii/features/society/2012/10/16/feature-01) aboard the [Hacker Bus](http://onibushacker.org/) in Brazil? + +3. Pushing the Limits of Engagement
+One of the great powers of social media is in its ability to break down traditional communication barriers. It also allows for a more informed discussion and interaction. Social media integration is just the start- what else can we do to push the limits of open communication, conversation, and engagement? Dr. Agnes Binagwaho, Rwanda's Minister of Health, illustrates the [power of social media coupled with senior official engagement, endorsement, and commitment](http://dr-agnes.blogspot.co.uk/2011/12/mondays-with-minister-3-malnutrition-in.html). With growing demand for financial demand from the public [see new civil society driven efforts like Open Budget Morocco](http://floussna.ma/), it is increasingly important to meet this interest with smart engagement and supply. +Are we off base? Would you like to add or highlight other areas? Have relevant experiences or insight to share? Let's talk. Leave comments below or tweet [#WBFinances](https://twitter.com/search?q=#wbfinances&src=savs). +
+##Big Dreams, Small Steps
+Imagine if all the pieces fall into place. Mobile technology continues to make leaps and bounds and becomes more accessible and affordable. Social media extends its reach and becomes as common as communicating by telephone or in person. Open Data is embraced by all public institutions. Current barriers to access to public information in effect would cease to exist. It would no longer be about who has data, but more importantly what one can do with data- a beautiful world, isn't it? The possibilities for good are endless. +Many development agencies and NGOs are thinking hard about and working on closing the "feedback loop," but a more ambitious scenario would be to eliminate the need for a feedback loop altogether. Imagine being so embedded and engaged in communities of interest that the feedback would be constant and in real-time. + +But we don't live in an ideal world, and the promise that many of these ideas may not bear fruit the way they are envisioned to. However, this dream scenario is useful when we think of the intermediate steps along the way to reaching "development utopia." +Open Data is like a map. Coupled with mobile technology and social media, it may show us where we are and help us determine where we would like to go. Having this map is not enough, as we still need to get to our destination. + +So let's dream big, but take small steps together. + +--------------------------------------------------------------------------------------------------------- +Links - World Bank finances app:
+[Android](https://play.google.com/store/apps/details?id=com.worldbank.finances&hl=en)
+[iOS/iTunes](https://itunes.apple.com/us/app/world-bank-finances/id465555488?mt=8)
+[Web](http://financesapp.worldbank.org)
+[Website](https://finances.worldbank.org)
diff --git a/examples/openspending/content/blog/2013-02-12-open-spending-in-morocco.md b/examples/openspending/content/blog/2013-02-12-open-spending-in-morocco.md new file mode 100644 index 00000000..6c83a82c --- /dev/null +++ b/examples/openspending/content/blog/2013-02-12-open-spending-in-morocco.md @@ -0,0 +1,105 @@ +--- +authors: +- martin +redirect_from: /2013/02/open-spending-in-morocco/ +title: Opening Budgets in Morocco +--- +This post is written by [Martin Keegan](https://twitter.com/mk270). + +In January I attended a conference in Morocco at the +[University of Mundiapolis in Casablanca](http://mundiapolis.ma/), to present OpenSpending. The +conference attracted a broad range of interests: civil society +organisations, journalists, private companies, academics and students, +the World Bank, and the director general of Morocco's Ministry of +Finance. My presentation ([notes here](http://mk.ucant.org/media/openspending-francais/)) showcased the use of OpenSpending in a variety of countries, by +state, civil and commercial actors. We are very generous to [Transparency +International Morocco](http://www.transparencymaroc.ma/) and the University +for their gracious invitation. + +The conference covered the efforts in the country, the region and globally +towards fiscal transparency and accountability; there was much discussion of +the proposed Freedom of Information law in Morocco, and the Moroccan +government's efforts towards publishing fiscal data. Several of the +presentations incorporated technology demonstrations. There was some +local press coverage of the conference, and some involvement from broadcast +media (in +French +[1](http://www.lnt.ma/economie/la-transparence-budgetaire-au-centre-dune-rencontre-debat-63825.html) +and +[2](http://www.leconomiste.com/article/902383-la-transparence-budg-taire-fait-d-bat)). + + +What maybe was not clear to the audience (and indeed, the press +coverage reflected this) was how OpenSpending and similar tools fit +into the landscape. I was directly asked "is the Government of +Morocco going to use OpenSpending?", and very firmly responded that +this was a question for the Government (after all, a representative of +the government was sat only a few paces away). The boundary between +government and private sector (both commercial and civil society) is +always evolving and sometimes contested, and it may be locate differently +in different societies. OpenSpending enables actors of all types (government, +private commercial, private non-profit and individuals) to participate +in discussions of spending, and by changing the costs can help enable +some actors to participate where previously this was infeasible. + +The key point why I made in response to this question is that a +Freedom of Information law can in theory substitute for +government-initiated publication of spending data, but that this +should be a complement, not a substitute. What I ought to have +remembered to say was that the access to information, either via FOI +or regular publication does not guarantee that the information be Open +Data, which is effectively necessary for civil society use in +OpenSpending. + +It was the experience of [MySociety](http://mysociety.org) that their parliamentary monitoring +tools were never accused of being partisan. It's important to keep +out of conventional politics: we're here to say "if you have transparency, +this is what you get" not "you should have transparency" or "you should spend +more money on this and less on that". The bulk of my presentation was on +four or five country studies, demonstrating the diversity of the places +which can use OpenSpending. + + + +The details of any national budget are a always a matter of healthy +political controversy, and this is [especially true of +Morocco](http://www.reuters.com/article/2012/11/18/us-morocco-protest-idUSBRE8AH0LX20121118). The Moroccan constitution contains an as yet unimplemented undertaking +to establish a Freedom of Information law. It still remains to be seen to what extent this will enable civil society in the future +to obtain more information about expenditures in the Moroccan national budget. In practise Freedom of Information laws will likely not do it alone. They tend often to allow the requester to specify the format of the material published, +or that it be published as open data. Where governments publish fiscal +information AND comply with an FOI law, it can be a [somewhat confrontational +move](http://constitution-unit.com/2011/05/24/we-can-work-it-out-eric-pickles-vs-nottingham-city-council/) to use the FOI laws to compel publication of fiscal data + +It was exciting to see that Transparency International Morocco have used the OpenSpending software +for [a site on the Moroccan budget](http://floussna.ma/). They however encountered some challenges with the implementation, which the OpenSpending community will work with them +to help resolve or mitigate these such as language. +By the stage in the conference where the Moroccan OpenSpending instance was +demonstrated, the audience had been subjected to about ten other countries' +spending visualisations, so this presentation by Dr Nesh-Nash came across +as just more of the same, which is an excellent situation to be in. The +audience immediately took to noting that regional spending was arguably +misallocated in his visualisation: this is great progress on the previous +state of not having easy access to this information, and underlies the point +that the raw data, and interpretations thereof, are linked but separable. + +[Samuel Lee from the World Bank demonstrated their new mobile app](http://openspending.org/blog/2013/01/29/worldbank-guest-post.html) and its Arabic language features, which was very well receieved from the participants. +Ultimately it would be great if OpenSpending could have an Arabic translation; it's +an official language in about twenty countries, many of which don't +use multiple languages as in Morocco. The public role +of OpenSpending to explain budgetary data depends on its ability to be read in the +languages people use. We're therefore pushing for the localization of OpenSpending into +French as well as another 15 languages. You can check our progress and help contribute at our site on the +[translation platform Transifex](https://www.transifex.com/projects/p/openspending/). + + +Several participants at the conference placed a particular emphasis on what was termed +"[vulgarisation](http://fr.wikipedia.org/wiki/Vulgarisation)", by which +is meant the business of describing and explaining information intellgible +to experts such that the general public can understand it as well (the word +lacks the pejorative connotations it possesses in English). What this really +means is that in French the motivation of OpenSpending is expressible in +a *single word*. We look forward to follow the developments on budget transparency in Morocco. The broad participation at this conference served to show that the interest will remain strong. + + + + diff --git a/examples/openspending/content/blog/2013-02-14-Join-hangout-on-the-EU-budget.md b/examples/openspending/content/blog/2013-02-14-Join-hangout-on-the-EU-budget.md new file mode 100644 index 00000000..5b836ba6 --- /dev/null +++ b/examples/openspending/content/blog/2013-02-14-Join-hangout-on-the-EU-budget.md @@ -0,0 +1,29 @@ +--- +authors: +- anders +redirect_from: /2013/02/Join-hangout-on-the-EU-budget/ +title: How to find spending data inside EU's budget - join us for a Google Hangout! +--- + + +
+
+Last week leaders from the European Union agreed, after months of haggling, on the 2014-2020 EU budget - also known as the Multiannual Financial Framework (MFF). The [€960bn budget](http://openspending.org/eu-budget-mff/entries) shown above marked a [widely reported reduction](http://www.bbc.co.uk/news/world-europe-20392793) in the budget, but many other stories can be told from the [48 page agreement (PDF) ](http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ec/135344.pdf). Despite accounting for merely [1.08 % of EU's GNI](http://europa.eu/newsroom/highlights/multiannual-financial-framework-2014-2020/index_en.htm), the EU budget should arguably be considered important for every citizen or journalist with a spending nerd hidden inside. Whether you think of [the last minute cuts in broad band investments](http://www.guardian.co.uk/technology/2013/feb/11/broadband-budget-cut-rural-connection-billion-euro) or the continued commitment to one of the worlds largest [farm subsidy programmes](http://farmsubsidy.org/), the agreement on the EU budget reflects the current priorities of the member states. + +At OpenSpending we want to invite journalists and the OpenSpending community to share experiences about how to access and analyse the spending data behind EU's budget.
+ +Join us Monday 18 February at 19:00 CET / 18:00 GMT for a Google Hangout. +[Ronny Patz](http://twitter.com/ronpatz) from [Transparency International in Brussels](http://www.transparencyinternational.eu/) will join and share his experiences on access to information on EU spending. + +At the hangout we'll discuss how to get started if you want to report on the spending behind the EU budget, and attempt to answer: +
  • Where to find EU spending data?
    +If you have decided to investigate EU spending, you want to locate who is responsible for releasing the data. We will talk about how to find spending data published at both [EU-level](http://ec.europa.eu/beneficiaries/fts/index_en.htm) and [national level](http://eustructuralfunds.gov.ie/). We have gathered a lot of information in this [map of EU spending](http://openspending.org/resources/eu/index.html).
    • + +
  • What's new to report on in the 2014-2020 EU budget?
    +How do you examine the consequenses to smaller programmes such as [nature conservation](http://rendezvous.blogs.nytimes.com/2013/02/11/e-u-says-20-percent-of-budget-green-critics-disagree/)? +And how do you detect changes in the distribution of the structural funds?
    • +
    +If some of the questions above have peaked your interest you should join us at the hangout. If you wish to get started now, have a look at some of the [EU spending data](http://openspending.org/eu-commission-fts) or [notes](http://openspending.org/resources/eu/notes.html) we're already featuring. +Join the Google Hangout by adding your name and email in the form below. + + diff --git a/examples/openspending/content/blog/2013-02-22-OpenSpending-around-the-world.mdown b/examples/openspending/content/blog/2013-02-22-OpenSpending-around-the-world.mdown new file mode 100644 index 00000000..01106eff --- /dev/null +++ b/examples/openspending/content/blog/2013-02-22-OpenSpending-around-the-world.mdown @@ -0,0 +1,48 @@ +--- +authors: +- anders +redirect_from: /2013/02/OpenSpending-around-the-world/ +title: OpenSpending around the world, Week 8 +--- + + + +We're bringing you an update with news from OpenSpending and financial transparency around the world. + +### Fresh data and activities from OpenSpending +City budgets: We're thrilled to see how lots of new cities have been added to OpenSpending. Have a look at some of the budgets added recently from [Umeaa, Sweden](http://openspending.org/budget2013politicalview/views/budgetdata-2103-nonpolitical), [St. Etienne, France](http://openspending.org/budget_2013_saint-etienne/views/budget-2013-de-saint-etienne), [Laatzen, Germany](http://openspending.org/stadtlaatzenplan2013), [Rotterdam, The Netherlands](http://openspending.org/begrotingrotterdam2012_3?_view=default6) and [Oakland, United States](http://openspending.org/oakland_adopted_budget_fy2011-12/views/city-of-oakland-adopted-budget-fy-2011-12-dept-program-expenditures-tree-map). + +From the 135,000 person town of Cary, United States, you can now browse [expenditures](http://openspending.org/town_of_cary_expenditures/views/expenditures-fy2011) and [revenues](http://openspending.org/town_of_cary_revenues/views/revenue-fy2011) on everything from the [fire services](http://openspending.org/town_of_cary_expenditures/Program/27/entries#Program:27) to [solid waste management](http://openspending.org/town_of_cary_expenditures/Program/52/entries#Program:52) from 2009-2011. + +Australia: Data on more than 5,000 [Federal government contracts](http://openspending.org/australian_federal_government_contract_spending/entries) from 2010-2011 has now been added. + +EU: [We covered](http://openspending.org/blog/2013/02/14/Join-hangout-on-the-EU-budget.html) the newly passed budget for the European Union for 2014-2020 with a GoogleHangout. In the future we plan follow up with more coverage on access to EU spending data. + +Morocco: Last month [Martin Keegan](https://twitter.com/mk270) did a talk on OpenSpending at a conference on transparency in Morocco. Read his take from the event [here](http://openspending.org/blog/2013/02/12/open-spending-in-morocco.html). + +### Spending transparency around the world +Earlier this month the Sunlight Foundation (USA) relaunched [Clearspending](http://sunlightfoundation.com/clearspending/), with updated transactional spending data from the United States federal government for 2012. A thorough [review](http://sunlightfoundation.com/blog/2013/02/04/clearspending-released-with-new-data/) of the new data found $1.55 trillion in misreported grants. + +Haiti: A [report](http://www.cgdev.org/doc/full_text/CGDBriefs/1426965/US-Spending-in-Haiti-The-Need-for-Greater-Transparency-and-Accountability.html) from the Global Center for Development concluded that tracking of US disaster relief spending in Haiti has been nearly impossible due to lack of transparency of NGOs and constractors. The report recommends to require prime contractors to report subcontracting data. + +The Comptroller of New York City [relaunched](http://techpresident.com/news/23404/new-york-city-officials-announce-new-dashboard-municipal-spending) its transparency site [Checkbook NYC](http://www.checkbooknyc.com/spending_landing/yeartype/B/year/114) with transactional spending data from across departments. The site offers access to data both via download and API. + +### Resources +A new study [Assessing Open Government Budgetary Data in Brazil](http://www.gpopai.usp.br/IMG/pdf/Craveiro-ICDS2013.pdf), has been published by Gisele S. Craveiro and her colleagues at University of São Paulo. The comprehensive paper reviews data from 54 budgetary websites across different Brazilian executive power levels (national, state and municipal), as well as 34 Brazilian audit court websites. + +[Sam Lee](https://twitter.com/OpenNotion) from the World Bank reviewed in a [guest post](http://openspending.org/blog/2013/01/29/worldbank-guest-post.html) how mobile technologies can help advance financial transparency. + +The International Budget Partnership published the [Open Budget Survey](http://survey.internationalbudget.org/), with detailed data on budget processes across 100 countries. Following the launch we hosted a community call discussing the survey. + +The newest issue of [Public Deliberation](http://www.publicdeliberation.net/jpd/) is dedicated entirely to participatory budgeting and include several articles worth a read. + +### Events on our radar +Open Data Day is coming up tomorrow Ferbruary 23. Join an [event near you](http://opendataday.org/)! + +On March 6 journalists who enjoy to dig deep into the rules of the #EUbudget are in for a treat, when the European Journalism Center organise a seminar dedicated solely to [new rules for spending in EU programmes](http://www.ejcseminars.eu/index.php/seminars/350/new-rules-for-the-unions-budget-simplified-access-to-funding-better-accountability-and-further-improvements-of-eu-spending-programmes). + +On May 2-4 the [Data Harvest](http://www.wobbing.eu/news/look-back-data-harvest-conference) will bring geeks, journalists and civi hackers together in Brussels to wrangle EU spending data from [farm subsidies](http://farmsubsidy.org/) and the [Comission](http://openspending.org/eu-commission-fts). + +We're hosting bi-monthly community calls, and are eager to hear your [ideas](https://twitter.com/openspending) for topics we should cover. + +How to get involved? Join the discussion on our [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2013-02-28-Budget-Stories.mdown b/examples/openspending/content/blog/2013-02-28-Budget-Stories.mdown new file mode 100644 index 00000000..3102bbba --- /dev/null +++ b/examples/openspending/content/blog/2013-02-28-Budget-Stories.mdown @@ -0,0 +1,19 @@ +--- +authors: +- victoria +redirect_from: /2013/02/Budget-Stories/ +title: Budgetstories.md - using open budget data to create meaningful stories +--- + + +This is a guest post by [Victoria Vlad](https://twitter.com/victoriavladd) from [BudgetStories.md](http://www.budgetstories.md/). + +Today [Expert-Grup](http://www.expert-grup.org/), an independent think tank based in Chișinău, Moldova, launched [BudgetStories.md](http://www.budgetstories.md/). [BudgetStories.md](http://www.budgetstories.md/) is an open budget website, which includes infographics, budget data visualizations and analysis of the use of public money in Moldova across sectors such as: the public administration, agriculture, education and health. In recent years Moldovan government has become more transparent regarding budget data as well as other types of data. The Ministry of Finance used the World Bank’s [BOOST tool](http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPUBLICSECTORANDGOVERNANCE/0,,contentMDK:23150652~pagePK:148956~piPK:216618~theSitePK:286305,00.html), to release detailed and disaggregated data on public expenditures. + +Since mid 2012, we’ve however worked to make sense and meaning in the huge sets of data. In November 2012, while finalizing the project concept, members of the organising team took part in the [Balkan Budget Workshop](http://openspending.org/blog/2012/11/26/Sarajevo-Workshop-Writeup.html) organised by the OpenSpending team. We decided to use OpenSpending as a tool for visualizing Moldova's [2013 budget](http://www.budgetstories.md/bugetul-2013/). + +Until now civil society (NGOs, journalists and universities) has shown little knowledge or interest in the existence of open data. Our target groups are mainly journalists and policy makers, who will now with this site for the first time have access to “cleaned data sets”. We are therefore hoping that they will republish and reuse the analysis and visualizations, which could trigger increased public attention to inefficiencies identified in government spending. Also, we’d like to expand the network of stakeholders who use the budget data and disseminate information about how the Government of Moldova spends public money. + +It is our hope that this project will create a better understanding among citizens and active members of the society about the way the public finance system operates and the way it influences their everyday lives. If we could reach such increased understanding this could ultimately lead to greater contributions from society to the budget process and more efficient spending of public money. + +In the future we plan to add interactive modules such as a real time budget calendar and a tax calculator. You can find out more information about BudgetStories.md on [Facebook](https://www.facebook.com/pages/Budget-Stories/572468439448024?sid=0.5174039560370147) and [Twitter](https://twitter.com/BudgetStories). BudgetStories.md is supported by [Soros Foundation Moldova](http://soros.md/) and [Open Society Foundations](http://www.opensocietyfoundations.org/). diff --git a/examples/openspending/content/blog/2013-03-01-OpenSpending-around-the-world-Week-9.mdown b/examples/openspending/content/blog/2013-03-01-OpenSpending-around-the-world-Week-9.mdown new file mode 100644 index 00000000..ef4ba387 --- /dev/null +++ b/examples/openspending/content/blog/2013-03-01-OpenSpending-around-the-world-Week-9.mdown @@ -0,0 +1,38 @@ +--- +authors: +- anders +redirect_from: /2013/03/OpenSpending-around-the-world-Week-9/ +title: OpenSpending around the world, Week 9 +--- + +### Fresh data and activities from OpenSpending +During Open Data Day we saw activities across cities.In Ottawa [Stephane Frechette](https://twitter.com/sfrechette) helped upload budget data for [city for 2013](http://openspending.org/ottawa_exprev_2013/views/treemap-city-of-ottawa-expenditure-revenue-summary-by-category-2013-estimate). + +In Goettingen, Germany the Pirate Party used OpenSpending to visualize the city budget for [expenditures and revenues](http://offenerhaushalt.piratenpartei-goettingen.de/goettingen-haushalt-2011.php?view=). + +[Victoria Vlad](https://twitter.com/Victoriavladd) from Expert Grup added budget data from [Moldova](http://openspending.org/sintezabugetuluidestat2013cheltuieli/views/vizualizare-sinteza-bugetului-de-stat-2013-pe-cheltuieli) and wrote a [guest post](http://openspending.org/blog/2013/02/28/Budget-Stories.html) yesterday about BudgetStories.md, a new project which is trying to explain budget data recently released in Moldova. + +As a growing number of city budgets are to OpenSpending we're looking to map these in a GoogleDoc, in order to visualise them on the site. You can help by adding the cities, which are already on OpenSpending in [this GoogleDoc](https://docs.google.com/spreadsheet/ccc?key=0AqR8dXc6Ji4JdHZZNUpWQ2paY3FfYTdFNXkxZXZDTWc#gid=0). + +On Thursday March 7, we're organising another Community Call, as Alan Hudson from ONE joins us to discuss: "How to use spending data to estimate unit costs and development outputs across the world?" +All details on the agenda and how to register to join are available [here](http://wdmmg.okfnpad.org/22?). + +### Spending transparency around the world +In the Miami Herald, [Nathaniel Heller](https://twitter.com/Integrilicious) of Global Integrity called the state of [financial transparency in Florida](http://www.miamiherald.com/2013/02/17/3237026/floridas-murky-fiscal-transparency.html#.USD_R5DCCvg.twitter) into question following an unsuccessful launch of a $5m transparency portal. + +During this week a one year old [blog post titled "Fiscal transparency (is not enough)"](http://davidsasaki.name/2012/04/fiscal-transparency-is-not-enough/) by [David Sasaki](https://twitter.com/oso), got the (again) from the budget geek community and sparked [this response](http://openbudgetsblog.org/2013/02/28/twitter-activism-is-not-enough/) from the OpenBudgetsBlog. Both posts raise releavnt questions about the importance of access to fiscal data and how to measure the release of such financial data. + +In a thorough local spending story Ars Electronica reports how a one-room library in West Virginia purchased a [$20,000 CISCO router](http://arstechnica.com/tech-policy/2013/02/why-a-one-room-west-virginia-library-runs-a-20000-cisco-router/). + +### Resources +The state of New Jersey (US) has produced a useful [guide for citizens (PDF)](http://www.state.nj.us/treasury/omb/publications/12citizensguide/pdf/citguide.pdf) to make the budget more accessible. + +### Events on our radar +March 7: OpenSpending Community Call with Alan Hudson. All details are available [here](http://wdmmg.okfnpad.org/22?). + +March 9: [A Hackathon in Ville de Gatineau, Canda](http://gatineauouverte-hackathon03-esli.eventbrite.ca/) will among other topics deal with OpenSpending. The budget from the city [is already up](http://openspending.org/gatineau_deprev_cat_12_13). + +May 2-4: [Data Harvest](http://www.wobbing.eu/news/look-back-data-harvest-conference) will bring geeks, journalists and civi hackers together in Brussels to wrangle EU spending data from [farm subsidies](http://farmsubsidy.org/) and the [Comission](http://openspending.org/eu-commission-fts). Information about how to register will follow. + +We're hosting bi-monthly community calls, and are eager to hear your [ideas](https://twitter.com/openspending) for topics we should cover. +How to get involved? Join the discussion on our [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2013-03-05-spending-stories-open-spending-are-moving-beyond-visualizations.mdown b/examples/openspending/content/blog/2013-03-05-spending-stories-open-spending-are-moving-beyond-visualizations.mdown new file mode 100644 index 00000000..50e19861 --- /dev/null +++ b/examples/openspending/content/blog/2013-03-05-spending-stories-open-spending-are-moving-beyond-visualizations.mdown @@ -0,0 +1,31 @@ +--- +authors: +- lisa +redirect_from: /2013/03/spending-stories-open-spending-are-moving-beyond-visualizations/ +title: Spending Stories, Open Spending Are Moving Beyond Visualizations +--- + + + +This post is by [Lisa Evans](https://twitter.com/objectgroup) of the Spending Stories project at OpenSpending and cross-posted from the [PBS Idea Lab](http://www.pbs.org/idealab/2013/03/spending-stories-open-spending-are-moving-beyond-visualizations059.html). + +2013 is going to be a big year for the spending stories project. In 2012, as we explained [in more detail on our blog](http://openspending.org/blog/2012/12/24/Roundup.html), we improved [usability of our platform for spending data](http://openspending.org/help/index.html) and developed stronger community ties around the world. Now we're primed to roll out some really empowering resources for the open spending community based on our experiences so far. Here's a look at what to expect in 2013. + +###Badge of approval +To build on our reputation as a trusted source of financial data we will introduce badges that show for each dataset uploaded that it has been approved and sanity-checked by our platform editors. So when you embed an Open Spending visualization on you website, you can be clear about the data quality, and when you [search Open Spending](http://openspending.org/search) you can, if you want to, include only approved sources and contributors. + +###New analysis engine +Up until now, Open Spending has largely been considered as a visualization tool. While we hope to see the data displays continue to thrive, there's a lot more under the bonnet that we'd like to bring to the front. With the U.K. in a period of austerity, it's quite possible the transaction data can shed some light on this highly sensitive topic. As such, we are planning a new analysis engine for financial transactions that will, among other things, show the biggest suppliers and show the results in insightful visualization tools. + +The analysis engine also will feature long-awaited new visualizations for analysis and presentation of the data, such as time-series views and bar charts -- handy for spotting things like spending patterns and cost overruns which could signal inefficiency. Visualizing the transaction data will help you with things such as tracking a supplier, tracking a department, and comparing departments and suppliers, to name but a few. + +###Real-time alerts +Via [Journoid](https://github.com/pudo/journoid) we plan to provide real-time alerts for local journalist or activists when new financial data arrives. We are introducing a pilot on the U.K. transaction data, giving new payments as they are published by central government departments or local municipalities. + +###Data expeditions +Last year we held a number of workshops that helped people tell stories with data. Drawing on the success of the data expedition at Mozfest in London in November, we are designing online and offline spending [data expeditions](http://blog.okfn.org/2012/11/14/data-expeditions-at-mozfest/) using spending and budget data -- starting with a series of courses taking you from fact checking to the final expeditions where you'll be comparing and analyzing data. + +###Data-driven investigations +Countries like [Brazil](http://openspending.org/blog/2012/07/19/Caring-for-your-neighbourhood.html) and the U.K. have begun releasing detailed spending data on contractors. We're looking to enable journalists as well as CSOs and academics to conduct data-driven investigations on spending, in particular transactions. Mentors will assist community members to start working with spending data and sharing methods for how to analyze and visualize these. We will use events and online training sessions to get feedback and advice on how to design the tools needed for scaling investigations of transaction data, which today remain vastly unexplored. + +We are very excited about a future filled with more real-time, fine-grained spending data, in which governments are really doing their bit to be accountable -- and we are responding by intelligently analyzing their open data. Together with local newsrooms, NGOs and dedicated citizens around the world, we will work together to map the world's spending data. diff --git a/examples/openspending/content/blog/2013-03-13-Launching-the-Aid-Transparency-Tracker.mdown b/examples/openspending/content/blog/2013-03-13-Launching-the-Aid-Transparency-Tracker.mdown new file mode 100644 index 00000000..4f409e83 --- /dev/null +++ b/examples/openspending/content/blog/2013-03-13-Launching-the-Aid-Transparency-Tracker.mdown @@ -0,0 +1,106 @@ +--- +authors: +- mark +redirect_from: /2013/03/Launching-the-Aid-Transparency-Tracker/ +title: Publish What You Fund Launches Aid Transparency Tracker +--- + +Publish What You Fund has undertaken some initial [analysis of aid donors’ plans](http://tracker.publishwhatyoufund.org/) to publish to the [IATI](http://www.aidtransparency.net/) component of the agreed [common standard](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) for aid information. Here, [Mark Brough](https://twitter.com/mark_brough) explains the process they went through to take a series of Excel files, convert them into a format suitable for analysis, and come to some overall conclusions. + +The short version: check out the [Aid Transparency Tracker](http://tracker.publishwhatyoufund.org/). + +###Why we did this +At the 4th High Level Forum on Aid Effectiveness in Busan, South Korea, in November 2011, the world’s aid donors agreed a “common, open standard” for publishing aid information. Donors also agreed in Busan to publish “implementation schedules” explaining in detail how and when they would meet this commitment. + +The implementation schedules are like a forward-looking calendar, explaining when donors plan to publish specific pieces of data, like results, project documents, and conditions, as well as transaction-level spending data. They also explain whether the donor will be publishing under an open license (public domain or attribution-only) and whether they will be republishing every quarter as a minimum frequency of publication – both are required for IATI compliance. These implementation schedules were published on the [OECD/DAC’s common standard website](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm), but in several different formats, which required a detailed look at each donor’s schedule, as well as interpreting them when donors have completed them in different ways or understood the various options differently. + +Pulling all the schedules into a single application allows us to [assess donors’ overall ambition](http://tracker.publishwhatyoufund.org/organisations/); [compare fields across schedules](http://tracker.publishwhatyoufund.org/fields/); show the publication of [fields over time from different donors](http://tracker.publishwhatyoufund.org/timeline/); and provide [CSV](http://tracker.publishwhatyoufund.org/organisations/GB-1.csv), [JSON](http://tracker.publishwhatyoufund.org/api/publishers/GB-1/)/[JSONP](http://tracker.publishwhatyoufund.org/api/publishers/GB-1/?callback=callback) and [iCal feeds](http://tracker.publishwhatyoufund.org/organisations/GB-1.ics) for each donor. + +###The original implementation schedules +The implementation schedules were published in individual Excel files, containing three main sheets: general, agency, and activity-level information. + +
    General
    +Approach to publication: includes timeliness and frequency, licence, initial publication dates, scope of publication.
    + + + + +
    + + +
    + +
    Agency
    +Agency-level publication: includes country budgets, organisation documents.
    + + + + + +
    Activity
    +Activity-level publication: includes information about when the organisation will (or won’t) be compliant with each field in the standard.
    + + + + +
    + + +
    +
    +###Templates +While there is a template for the common standard implementation schedules, several different templates exist, and donors further added to this complexity by modifying the templates, changing options, and adding and deleting rows – in all, we counted eleven different versions. Importing the schedules proved difficult because of this. In addition, some donors published their Excel-based templates in PDF format, which made it impossible to automatically parse them. In these cases, it was necessary to copy and paste the same data into new spreadsheets. While the data was thoroughly checked, it is possible that some human error will have resulted. Some interpretation was necessary to ensure consistency and comparability across the schedules. + +###Importing +The schedules were automatically parsed and imported into this application. Publish What You Fund staff then checked the resulting data to ensure that it had been parsed correctly and that it made sense. + +
    Select a file
    +The user can select a schedule from any publicly-accessible URL. This could be the [OECD/DAC common standard](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) website, the [IATI website](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm), or the donor’s own website. Where the original schedules could not be automatically parsed (e.g. because they were in PDF), a new spreadsheet was created by Publish What You Fund using the information included in the original schedule. + + + + +
    + + +
    + +
    Check fields
    +The schedule was successfully parsed and is presented to the user to check and correct, make sure that all information has been parsed correctly, ensure that compliance status is consistent with notes, and score for partially compliant fields if the publisher has understated their publication relative to other donors. +The user can select a schedule from any publicly-accessible URL. This could be the OECD/DAC common standard website, the IATI website, or the donor’s own website. Where the original schedules could not be automatically parsed (e.g. because they were in PDF), a new spreadsheet was created by Publish What You Fund using the information included in the original schedule.
    + + + + +
    + + +
    + +###Openness begets openness +None of this would have been possible without:
    +1) A range of great open-source tools: +- [iati-implementationxml](https://github.com/Bjwebb/iati-implementationxml) (originally converted Excel IATI implementation schedules to XML) +- OKFN’s [ReclineJS](http://reclinejs.com/) (for the graphs and timeline; which itself relies on a range of great open source software) +- [Bootstrap](http://twitter.github.com/bootstrap/), [JQuery](http://jquery.com/), and [JQuery Tablesorter](http://tablesorter.com/docs/) (for the front-end) +- [Python](http://python.org), [Flask](http://flask.pocoo.org/), and a range of libraries (for the back-end)
    +2) [Implementation schedules](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) released by donors as part of the common standard, transparently outlining in detail their specific commitments to publish more and better data. + +But, we’ve also fed back: iati-implementationxml has been expanded to add compatibility with the common standard formats of implementation schedules; to detect the format of different schedules; and to function as a module that can be imported. We provided a couple of pull request for some small bugs with ReclineJS, and of course, we’re releasing all of our own code as well (although, it could definitely do with some tidying up…). + +###What’s next? +Now that we’ve looked at what information different donors are committing to publish, there are two main steps for us:
    + +1) Doing it again: we want to check the assumptions and interpretations we’ve made with donors to ensure that they’re accurate, as well as encouraging all donors to be more ambitious about the information they’re planning to publish. As part of this, we’ll also aim to iron out the problems in the schedule.
    +2) Looking at publication: now that we have better data on what donors are committing to do, the question arises: what are they actually publishing, and are they meeting their commitments? We’ll be working on that throughout the year and will release this analysis in October. Until then, all of our code is of course being developed as an [open-source project on Github](https://github.com/markbrough/IATI-Data-Quality). +
    +
    We’d love your feedback, so please get in touch:
    +[email](mailto:info@publishwhatyoufund.org)
    +twitter: [@mark_brough](https://twitter.com/mark_brough) or [@aidtransparency](https://twitter.com/aidtransparency) + +###Links +[Aid Transparency Tracker](http://tracker.publishwhatyoufund.org/)
    +[Source code](https://github.com/Bjwebb/iati-implementationxml)
    +Licenses: [AGPL v3](http://www.gnu.org/licenses/agpl-3.0.html) (code); [PDDL](http://opendatacommons.org/licenses/pddl/) (data); [CC-BY](http://creativecommons.org/licenses/by/3.0/) (report and analysis)
    +[IATI (International Aid Transparency Initiative)](http://www.aidtransparency.net/)
    +[OECD/DAC Common Standard website](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) diff --git a/examples/openspending/content/blog/2013-03-15-OpenSpending-around-the-world-Week-11.mdown b/examples/openspending/content/blog/2013-03-15-OpenSpending-around-the-world-Week-11.mdown new file mode 100644 index 00000000..6a5d8e1f --- /dev/null +++ b/examples/openspending/content/blog/2013-03-15-OpenSpending-around-the-world-Week-11.mdown @@ -0,0 +1,45 @@ +--- +authors: +- anders +redirect_from: /2013/03/OpenSpending-around-the-world-Week-11/ +title: OpenSpending around the world, Week 11 +--- + +We're bringing you an update with news from OpenSpending and financial transparency around the world. + +### Fresh data and activities from OpenSpending +[Tajima Itsuro](https://twitter.com/niryuu) has written an [excellent guide](http://qiita.com/items/4adb658c627d2e6d48e4) in Japanese for uploading data to OpenSending. + +The [Open Data Census](http://census.okfn.org/) demonstrated that several countries like Slovenia, Czech Republic and Italy (for EU Structural Funds) have released transactional spending data, which yet have to make it onto OpenSpending. If you can to help to get the data up on OpenSpending, we've added links and other information in this [GoogleDoc](https://docs.google.com/spreadsheet/ccc?key=0AvdkMlz2NopEdElqWTBJS0Q1Q083VlI3YUFLTl9OY0E&usp=sharing). + +We're also working to visualise the cities who are already on OpenSpending. You can help by adding cities in this [GoogleDoc](https://docs.google.com/spreadsheet/ccc?key=0AqR8dXc6Ji4JdHZZNUpWQ2paY3FfYTdFNXkxZXZDTWc#gid=0). + +Last week we had a community call with Alan Hudson from ONE, where we discussed: "How to use spending data to estimate unit costs and development outputs across the world?" +The agenda from the call is available [here](http://wdmmg.okfnpad.org/22?) and we'll follow up with more information about the ideas we discussed. + +### Spending transparency around the world +Health spending: Back in January Steven Brill published the long read article [The Bitter Pill](http://www.time.com/time/magazine/article/0,9171,2136864,00.html), in which he documented the lack of pricing transparency on the American health care market. The story led to this [podcast on Planet Money](http://www.npr.org/blogs/money/2013/02/26/172996963/episode-439-the-mysterious-power-of-a-hospital-bill), where they discussed the key sources of the piece - the medical bill and the untransparent pricing at American hospitals. + +EU Structural funds: Consultant and analyst [Luiggi Reggi](http://www.luigireggi.eu/) has [mapped the various formats member states](http://www.luigireggi.eu/Innovation-policies/Home/Entries/2012/11/9_an_interactive_map_to_find_real_open_data_on_Structural_funds_ACROSS_EUROPE.html), when publishing spending data from the EU Structural Fund at national and regional sites. The study shows that Italy and Poland are among the countries delivering machine redable CSV-files, whereas countries like Germany and Spain only release data in PDF. + +EU fish subsidies: Last month the European Commission published a report highlighting the [poor quality of data on fish subsidies and over fishing](http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0085:FIN:EN:PDF). About Spain, the biggest receiver of fish subsidies, the report concluded: "A comparable and objective evaluation of overfishing and of economic sustainability is not +possible due to absence of data." + +Tax app: In Philadelphia (US) the initiative to reassess property taxes in the city spured the development of an app documenting changes in tax payments [parcel by parcel](http://axisphillyapps.tumblr.com/post/44714283089/how-we-made-the-avi-map). + +### Resources +Marc Maxson used the model of [Benford's Law](http://en.wikipedia.org/wiki/Benford's_law) to analyse financial data from a few [public agenices in Kenya and an international non-profit](http://chewychunks.wordpress.com/2013/02/17/the-weekend-i-audited-the-world/). Benford's Law can be used to assess distributions across financial transactions to detect irregularities, which can be an indicator of corruption. + +Pulbish What You Fund launched this week a comprehensive Aid Transparency Tracker. [Mark Brough](https://twitter.com/mark_brough) explains how it works [here](http://openspending.org/blog/2013/03/13/Launching-the-Aid-Transparency-Tracker.html). + +### Events and deadlines on our radar +March 18: Deadline for proposals for the [News Challenge](https://www.newschallenge.org/) of the Knight Foundation. + +March 19: OpenData "Maker" night in London. OpenSpending will be looking into UK government finances at the. RSVP [here](http://bit.ly/Zpo5OU) + +March 25: Deadline for proposals for [Journalism Grants](journagrants.org) on international development issues. + +May 3-4: [Data Harvest](http://www.journalismfund.eu/dataharvest13) will bring geeks, journalists and civi hackers together in Brussels to wrangle EU spending data from [farm subsidies](http://farmsubsidy.org/) and the [Comission](http://openspending.org/eu-commission-fts). We'll be there and regsistration is now open. + +We're hosting bi-monthly community calls, and are eager to hear your [ideas](https://twitter.com/openspending) for topics we should cover. +How to get involved? Join the discussion on our [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2013-03-19-AfricanSpending-Knight-News-Challenge.mdown b/examples/openspending/content/blog/2013-03-19-AfricanSpending-Knight-News-Challenge.mdown new file mode 100644 index 00000000..ab53ec0a --- /dev/null +++ b/examples/openspending/content/blog/2013-03-19-AfricanSpending-Knight-News-Challenge.mdown @@ -0,0 +1,94 @@ +--- +authors: +- lucy +redirect_from: /2013/03/AfricanSpending-Knight-News-Challenge/ +title: AfricanSpending - Knight News Challenge with Open Institute and African Media Initiative +--- + +The Open Knowledge Foundation, in partnership with the Open Institute in Kenya and the African Media Initiative have submitted a proposal to the Knight News Challenge on Open Government with the title AfricanSpending - Monitoring the Money. The focus of this proposal is enabling journalists & CSOs to effectively track government finances. + +We want to develop [AfricanSpending](http://africanspending.org/) as a community driven project to monitor the money in Africa, building on the strong base we have already with OpenSpending and pulling in new types of data, very relevant for Africa... + +You can read the full proposal on the [Knight News Challenge Website](https://www.newschallenge.org/open/open-government/submission/africanspending-monitoring-the-money-enabling-journalists-csos-to-track-government-finances/) and if you like it - please applaud it! From tomorrow, you will also be able to provide feedback on the proposals. + +Read more on the proposal below! + +## Summary +We’ll build a community-driven platform with data resources (leveraging OpenSpending) for journalists and civil society to track public money and mineral wealth, plus related contracts and services, across Africa to combat cronyism and corruption. + +## In a nutshell + +

    AfricanSpending from Open Knowledge Foundation on Vimeo.

    + +Tracking and monitoring government finances - including those related to mineral wealth - are a major issue across Africa and much of the rest of the Global South. Journalists, civil society organizations and citizens, could and should play a large role in holding Governments to account by following the money. + +Unfortunately, African journalists and citizen groups seldom have the fiscal insight or technical skill to “map the money”. As a result, media coverage and public debate is shallow, reactive, and often fails to hold government to account or tell citizens how government action impacts their personal and local lives. + +This project is about dramatically improving this situation. + +We propose three key aspects of the project work: + +### 1. The Technology Platform + +We'll customize and extend the functionality and ease-of-use on the www.OpenSpending.org platform, to better track Government money and contracts across Africa. Being able to cover government activity related to the extractive industries and mineral wealth is key. Improving the way that OpenSpending analyses contracts, so that agreements and money flows can be linked, will therefore be a central focus. + +We'll also work to improve the ability to link money to people and organizations, enhancing and developing the existing work linking OpenSpending and OpenCorporates, as well as our existing work to link newsroom platforms such as [Document Cloud](www.DocumentCloud.org) and [Poderopedia](www.Poderopedia.org). + +Third, we’ll work to improve the relevance and accessibility of the resulting data that will allow citizens to compare the ‘real world’ value of expenditure or contracts, across regions or cities, or between planned and actual expenditure (we can take inspiration here from work like [GM’s Carbon Footprint toolkit](http://visualization.geblogs.com/visualization/co2/#/boiling_water_gas). + +By building on the existing OpenSpending platform, and its world-wide user-group, we'll be able to leverage existing technology and ensure our work benefits not just Africa or this project, but a global community. + +### 2. Getting The Data + +Whilst the platform will make it much easier for journalists and others to access and understand financial information, it will be of little value if it contains little or no data! A second element of our work will therefore be dedicated effort to obtain and process key Government financial information from as many countries as possible. We've already done substantial work here (Kenya, South Africa, Tanzania, etc), and run regular DataLiberation Scraperthons and [dBootcamp workshops](https://ghana.databootcamp.org) through the 13 African HacksHackers.com chapters that form part of our network. +
    + +
    +We also have working relationships with groups with relevant knowledge and skills elsewhere (e.g. Nigeria BudgIT and Revenue Watch on extractives, etc) and are assisting the 14-member African Network of Centers for Investigative Reporting (ANCIR) with their investigation into the African extractives industry. + +### 3. Education, Community Building & Engagement + +A platform and data have *no value* if they are not used. The final, essential, part of our work will be developing the awareness and capacity in key communities of journalists, CSOs and civic coders. To do this we will: + +* Build on what’s there: we are already the lead organizer for the HacksHackers community in Africa (2,000 plus members in 13 chapters) and the [Code for Africa initiative](http://beta.code-africa.org/), that coordinates the largest open data and open government initiatives in Africa through partnerships with the African Media Initiative, World Bank, and Google. We also have good connections with other CSOs working on finance, transparency and extractives across the continent +* Offer immersive training in the form of ‘Spending Bootcamps’. Hit and run training has little impact. Our bootcamps, modelled on our existing and successful dBootcamps, will therefore be structured as an investigative process that runs over three months each and helps participants build multi-disciplinary teams (of journalists, technologists, and CSO experts), find data (through DataLiberation Scraperthons), and then build projects (at the dBootcamps) that can be deployed in the real world -- all while learning to use new tools. +* Manage a [lightweight] fellowship programme: in environments with severe skills and resource challenges, you need champions who can serve as catalysts and ‘enablers’ to help kickstart mass uptake of new tools or resources. We will therefore run a fellowship programme for 12 annual Spending Fellows, who will initially spend three months each with the core OpenSpending team for intense hands-on training and mentorship, whereafter they will return to be embedded into thought-leader media and civil society organisations. Their focus will be to produce compelling journalism and meaningful civic engagement initiatives from spending data. Going beyond geek tools, we will stress pragmatic ways to demystify budgets and to give “actionable information” that ordinary citizens understand and care about. + +## Who Are Our Target Audiences? + + + +We’ll serve four audiences: journalists who want to use the site to improve the way they report on government activity, civil society organisations who want the tools and information to run “evidence-based” campaigns, civic hackers who want to use our data or resources to build thier own tools to improve government and empower citizens, and, finally, ordinary citizens who want easy access to “actionable” and customisable versions of their country’s spending information. + +We’ll reach the journalists through the growing network of Hacks/Hackers chapters across Africa. There are currently 13 of 20 planned chapters, with roughly 2,000 active members, who meet at least once monthly for skills exchanges and collaborative projects. We’ll reach newsrooms, civic hackers and civil society organisations through the Code for Africa initiative, which uses country-based initiatives such as www.Code4Kenya.org to embed data wranglers into media and NGOs with support from an external civic tech lab to help improve the use of digital tools and data resources. + +We’ll also bring together our existing networks topic specific experts in the NGO world, such as the local partners of the International Budget Partnership, with journalists and media organisations to help bring topic-specific expertise together with storytelling ability and develop ongoing relationships to help the data flow between organisations. + +## Who Are the Partners? + +AfricanSpending is a consortium of strategic partners, all with proven records for delivering on data and civic engagement initiatives, including: + +The African Media Initiative (AMI), which is an industry umbrella association of 600+ of Africa’s largest media companies. AMI currently runs a series of digital innovation programmes, investing almost $2 million annually into supporting digital and data initiatives in newsrooms on the continent. AMI also spearheads the Code for Africa initiative, building active citizenry and open data that goes beyond just open government. AMI will drive the media engagement component of AfricanSpending. + +The Open Institute (OI), a Kenya-based think/do tank that specialises in implementing open data and open government initiatives. OI is currently the lead implementing agency on Code4Africa on behalf of the World Bank and AMI, as well as for AMI’s dBootcamp data workshops, and on aspects of its www.AfricanNewsChallenge.org programme. OI will be the lead implementer on AfricanSpending. + +The Open Knowledge Foundation (OKF), the originator of OpenSpending and an international leader on open data tools with extensive experience in building dynamic communities and public engagement around public data. OKF will supply the technical platforms for AfricanSpending, and will host the African fellows at its hubs in London & Berlin. + +## The AfricanSpending Fellowship Programme + +Fellows will be selected through a competitive public process, and will be expected to return to their media and/or civil society organizations as both ambassadors and peer-mentors. Fellows will be trained to upload and manage data on the AfricanSpending platform, as well as how to build new engagement tools and visualizations based on local needs. + +## What Have We Already Built? + + + +The technology base for this project (OpenSpending) is mature and has been extensively used. In addition Open Knowledge Foundation, Open Insitutite & AMI / Code4Africa have already done African-specific work including work in Cameroon and the prototype [“AfricaSpending”](http://africanspending.org/) using national budget data collected for Kenya, South Africa, Tanzania, and Uganda. Open Institute meanwhile manages an instance of CKAN as an umbrella public [data portal for Africa](www.AfricaOpenData.org), on behalf of AMI. The data portal, which is used for dBootcamp and other AMI skills programmes, is currently the largest open data source in Africa. + +All our code is open source, so will be easy for others in the broader Code4Africa ecosystem to reuse components in different environments, or to integrate our platform with others, such as the Freedom of Information (FOI) request tracking portals already being funded by AMI. + +## Support the project + +You can read the full proposal on the [Knight News Challenge Website](https://www.newschallenge.org/open/open-government/submission/africanspending-monitoring-the-money-enabling-journalists-csos-to-track-government-finances/) + and if you like it - please applaud it! From tomorrow, you will also be able to provide feedback on the proposals. + +Photo credits: David Keats, le Korrigan on Flickr diff --git a/examples/openspending/content/blog/2013-03-20-How-to-Embed-Open-Spending-Databases-to-Your-Own-Website.mdown b/examples/openspending/content/blog/2013-03-20-How-to-Embed-Open-Spending-Databases-to-Your-Own-Website.mdown new file mode 100644 index 00000000..5c4752b8 --- /dev/null +++ b/examples/openspending/content/blog/2013-03-20-How-to-Embed-Open-Spending-Databases-to-Your-Own-Website.mdown @@ -0,0 +1,53 @@ +--- +authors: +- lisa +redirect_from: /2013/03/How-to-Embed-Open-Spending-Databases-to-Your-Own-Website/ +title: How to Embed Open Spending Databases to Your Own Website +--- + +This post is by [Lisa Evans](https://twitter.com/objectgroup) of the Spending Stories project at OpenSpending and cross-posted from the [PBS Idea Lab](http://www.pbs.org/idealab/2013/03/how-to-embed-open-spending-databases-to-your-own-website078.html). + +The Open Spending platform is a hive of activity and packed full of colorful displays of spending and budgets from all over the world. Its aim is to help track government and corporate financial transactions across the world and present them in useful and engaging forms. + +For some time now, users have been able to [upload](http://openspending.org/help/data-loading.html) any of their own spending to the platform and make a range of visualizations, but now you can embed any of the visualizations on your own website. This means you can have the full interactive display on your site. + +### How to embed +Let's say you have chosen a visualization on the Open Spending platform; notice there's an "embed" button at the bottom right of the webpage. The red arrow points to the button below. + + +
    +Just click "embed" and you'll be presented with the code to embed on your website and some options for the size (in pixels) of the interactive. +
    + +
    +The rest is just cut and pasting this code extract into your site. + +So, if you're a journalist looking to add an interactive to your online news story, an NGO looking to show spending on your cause, or you work in a government department and want to show how you allocate funds, then you can add this code to your site. If you are unsure how to paste the code then contact your site administrator. + +The reason it's possible to embed code comes down to the Open Spending widgets. + +In very simplified terms a widget is a piece of code you can add into your webpage, and it pulls data -- in this case from the Open Spending database -- so you don't need to store datasets yourself. If you have any difficulties or questions about using the Open Spending widgets, then please don't hesitate to [contact us](http://lists.okfn.org/mailman/listinfo/openspending). If you're curious about how the widget code works, then here are three examples shown in jsfiddle where you can see the full web environment for the:
    +
    +[Bubble Tree](http://jsfiddle.net/vitorbaptista/jhaKT/)
    +[Tree Map](http://jsfiddle.net/vitorbaptista/RVdNt/)
    +[Aggregate Table](http://jsfiddle.net/vitorbaptista/mFVMv/) +
    +The three main visualizations we offer at the moment are listed below along with links to the open-source code on Github. +
    +### The bubble tree + +
    +Widget code for the bubble tree on Github is [here](https://github.com/openspending/openspendingjs/tree/master/widgets/bubbletree). +
    +### The tree map + +
    +Widget code for the tree map on Github is [here](https://github.com/openspending/openspendingjs/tree/master/widgets/treemap). +
    +### The table view + +
    +Widget code for the table view on Github is [here](https://github.com/openspending/openspendingjs/tree/master/widgets/aggregate_table). +
    +And it's likely more interactive displays will be available over time. Have fun with the widgets. Let us know how easy you find them to use and when you use them by posting a link to your site on the Open Spending mailing list. +
    diff --git a/examples/openspending/content/blog/2013-04-04-FAQ-Daily-Bread-in-the-UK.mdown b/examples/openspending/content/blog/2013-04-04-FAQ-Daily-Bread-in-the-UK.mdown new file mode 100644 index 00000000..8ac9cc5b --- /dev/null +++ b/examples/openspending/content/blog/2013-04-04-FAQ-Daily-Bread-in-the-UK.mdown @@ -0,0 +1,50 @@ +--- +author: $authornamehere +redirect_from: /2013/04/FAQ-Daily-Bread-in-the-UK/ +title: FAQ - The Daily Bread in the UK +--- + +The annual release of the [budget proposal](http://www.guardian.co.uk/news/datablog/2010/oct/18/deficit-debt-government-borrowing-data) from the UK government last month, once again sparked interest in the [Daily Bread calculator](http://wheredoesmymoneygo.org/dailybread.html), which was featured at the [Guardian](http://www.guardian.co.uk/uk/datablog/interactive/2013/mar/20/budget-2013-how-taxes-spent-interactive). + +This FAQ is intended to give you a quick walk-trough on how and why we made the calculations behind the Daily Bread. If you have any questions or comments, we will be thrilled to hear from you on the [OpenSpending list](http://lists.okfn.org/mailman/listinfo/openspending). + +### How does Daily Bread calculate the tax I pay? +We use the tax rates available at the [HM Revenue and Customs (HMRC)](http://www.hmrc.gov.uk/rates/it.htm) to calculate the tax you pay. For each year from 2000 to 2012, HMRC provides information about the amount of money you could earn before you had to start paying tax on it. In 2013-14 you can earn £9,440 before you are taxed. + +We then take factors (deductions), which will raise your non-taxable income limit even higher. For instance blind people receive an additional £2,100 to their £9,440 non-taxable income limit, which bring their non-taxable income to £11,600 (that's £2,160 + £9,440). + +Other factors, which increase your non-taxable earning are: +- if you are married +- if you have turned 75 years old +- if you are between 65 to 74 years old + +We then record the tax bands for each year to the present. Tax bands work as follows: if you earn up to £32,000 you pay 20% of your income in tax, if you earn up to £150,000 you pay 40% of that in tax. + +As we have now got the tax rate, we record your [national insurance contribution](https://www.gov.uk/national-insurance/overview) for each year. National insurance is an extra tax you pay so you are entitled to certain state benefits, including the [state pension](https://www.gov.uk/browse/working/state-pension). The contributions you pay depend on how much you earn and whether you're employed or self-employed. You stop paying National Insurance contributions when you reach state pension age. So we have in the script we have different levels of income and the national insurance you are required to pay. + +### Which Year does Daily Bread calculate taxes for? +The Daily Bread uses information from the National Statistics report of the [effects of taxes on household income](http://www.ons.gov.uk/ons/rel/household-income/the-effects-of-taxes-and-benefits-on-household-income/index.html) to calculate tax rates. Currently the calculations are based on the tax rates for 2010 as these are the most recent figures available from the Office for National Statistics. +We are therefore unfortunately not able to apply the tax rates for 2013-14 to The Daily Bread, even though the rates are available in our [TaxMan](http://openspending.org/blog/2012/08/05/introduction-to-the-taxman.html). + +### Why is incomes at £10,0000 at 4,621 despite having a £9,440 non-taxable income? +The Daily Bread include several other indicators in its calculation besides the deduction of the non-taxable income. + +According to the [Office for National Statistics](http://www.ons.gov.uk/ons/rel/household-income/the-effects-of-taxes-and-benefits-on-household-income/2010-11/index.html), who estimates the average taxes paid by various incomes brackets, the breakdown of the average amount of tax paid by someone earning £10,000 in 2010 is: + +- direct taxes: £1122 +- indirect taxes vat: £1318 +- indirect taxes tobacco: £407 +- indirect taxes alcohol: £90+112 +- indirect taxes motoring: £268+84 +- indirect remainder: £1220 +- Sums to: £4621 + +### How does Daily Bread workout how much my tax contributes to public spending? +We use figures from Country Regional Analysis (CRA) to calculate your daily contribution to public spending. The current calculations are based on data from CRA from 2010, in order to provide synchronized across the various calculations we do. You can find the CRA data we use on the [Datahub](http://datahub.io/dataset/ukgov-finances-cra/resource/52fad6e9-d5b3-4ae7-b5ec-88f921af1623) under the name “OpenSpending-compatible CSV”. + +### Where can I find the code behind Daily Bread? +The code for Taxman, which calculates taxes is on [Github](https://github.com/openspending/taxman) +See the introduction to Taxman [here](http://openspending.org/blog/2012/08/05/introduction-to-the-taxman.html). + +The code for [Where Does My Money Go](http://wheredoesmymoneygo.org/) is available [here](https://github.com/openspending/wheredoesmymoneygo.org). + diff --git a/examples/openspending/content/blog/2013-04-06-Visualising-Urban-development-data-at-UN-Habitat.md b/examples/openspending/content/blog/2013-04-06-Visualising-Urban-development-data-at-UN-Habitat.md new file mode 100644 index 00000000..8386863f --- /dev/null +++ b/examples/openspending/content/blog/2013-04-06-Visualising-Urban-development-data-at-UN-Habitat.md @@ -0,0 +1,23 @@ +--- +author: $authornamehere +redirect_from: /2013/04/Visualising-Urban-development-data-at-UN-Habitat/ +title: Visualising Urban development data at UN-Habitat +--- + +Explore Visualizations - Open UN-Habitat Transparency Initiative + +This is a guest post by [Pontus Westerberg](https://twitter.com/pontusw), [UN-Habitat](http://open.unhabitat.org) + +In September 2012, UN-Habitat, the UN agency responsible for promoting sustainable towns and cities, became the third UN organisation to publish data to the [International Aid Transparency Initiative (IATI)](http://www.aidtransparency.net/) standard. As the project manager for what we have called “The Open UN-Habitat Initiative”, I am really pleased that we have taken this important step in our journey to become a more open and transparent organisation. + +At the same time as we began [publishing IATI data](https://github.com/openaid-IATI/OPEN-UN-HABITAT), we also launched the [Open UN-Habitat website](http://open.unhabitat.org/), which will be the focal point for our open data and transparency work over the coming months and years. At the moment (April 2013) the site contains information about 115 UN-Habitat projects, equivalent to 70 per cent of our current projects. It includes project data required by the IATI standard as well as project descriptions, budgets and information about project partners. We are working to provide additional documents for each project. + +Obviously releasing this kind of data takes time. One of our biggest challenges is lack of digitization. Many of our documents have been archived in the database as scanned copies, which means that entering or transcribing those raw data has been difficult. Essentially, a lot of the information would have to be retyped, which is an extremely resource-intensive exercise. Currently we are refining internal procedures to ensure that data is entered into databases in a timely and complete fashion. + +The technology behind the site is all open source. We worked with the Dutch company Akvo and their [OIPA tool](http://oipa.openaidsearch.org/api/v2/docs/) to parse the IATI data. OIPA is a search tool built in Django by Zimmerman & Zimmerman, which enables IATI compliant datasets to be easily parsed. The front end was built by Kenyan web agency Verviant in Wordpress and finally the visualisations were created using the bubble charts developed by the Open Knowledge Foundation. All technology we used is available under a Creative Commons license on Github. + +However we don’t think that publishing information about the projects we are involved in is enough. By May 2013 we will release all the statistical data collected by UN-Habitat over the years, which will include data on more than 1,200 cities. Some data sets such as the number of urban and slum dwellers has been published before in different formats, while other data sets have never previously been published. Few of the data sets have been published as open data before. In addition to publishing the raw data, we are currently looking at ways of visualizing it – comparing different data sets and looking at changes over time, with the aim of making the site becoming a leading resource for urban development data. + +If you have any comments or thoughts, we would be glad to [hear them](http://open.unhabitat.org/contact/). + +Pontus Westerberg is project manager at the Open UN-Habitat transparency initiative. diff --git a/examples/openspending/content/blog/2013-04-09-procurement-hack-day.mdown b/examples/openspending/content/blog/2013-04-09-procurement-hack-day.mdown new file mode 100644 index 00000000..7f852ebb --- /dev/null +++ b/examples/openspending/content/blog/2013-04-09-procurement-hack-day.mdown @@ -0,0 +1,30 @@ +--- +author: $authornamehere +redirect_from: /2013/04/procurement-hack-day/ +title: Join us for Hack Day on EU procurement data May 2 in Brussels +--- + +Next month OpenSpending will be at the annual [DataHarvest](http://www.journalismfund.eu/dataharvest13), where investigative journalists and civic coders dig into data projects from across Europe. The DataHarvest takes it name from the annual release of payments to farmers under the Common Agricultural Policy analyzes by the friends at [Farmsubsidy](www.farmsubsidy.org). + +We wanted to follow up on the work on EU procurements, which could become a useful source for journalists. Opented.org did some of the ground work [scraping](http://opented.org/code-repositories/) and parsing of the data from the [European procurement register, TED](http://opented.org/getthedata/). + +We're therefore excited to host a Hack Day on procurement data on May 2 before the DataHarvest thanks to support from the [Mozilla OpenNews programme](http://www.mozillaopennews.org/). + +**Register [here](http://procurement-hack.eventbrite.com/#)!** + +### Why open up procurements? +Procurement data tended, for good reasons, to enjoy much less attention from journalists than for instance spending data. A major reason is the fact that contract information tend to be more about text than spreadsheets, but also because access to aggregate data has been limited. However with access to more than 100,000 public sector contracts annually from the [European procurement register](https://ted.europa.eu) originating from tiny municipalities to large government agencies, there are good reasons to explore if contracts can help fill out the blanks. What is needed is good data feeds, which can provide journalists with contract data beyond the single search options offered at TED. So how does contract data supplement other data sources? + +####When spending data is unavailable +Spending data is useful as it provides a clear paper trail of actual individual payments over time, unlike contract information, which often include an indication by the tme of contract award. However today, most European governments have yet to publish transactional spending data as it for instance practised in the [UK](http://openspending.org/ukgov-25k-spending) and [Slovenia](https://www.kpk-rs.si/en/faq/supervizor). When looking at pubic agencies outside government such as regions or municipalities, where a sizeable share of government spending is being executed, access to transactional spending data is yet less common. Therefore procurement data on contracts awarded from any public agency above the EU threshold at EUR 200,000, is often the only resort for a paper trail documenting public spending on particular contractors. + +####Getting a sneak peak into publicly owned companies +The public does mostly not have access to information about how publicly owned companies such as power utilitiesspend their money. It is however an often overlooked fact that EU procurement rules apply to all majority publicly owned companies. That is the reason why the public can access more than 500 contracts awarded by the Swedish state-owned Vattenfall in all countries of operation, such as [this contract awarded from their Berlin based company](http://ted.europa.eu/udl?uri=TED:NOTICE:12176-2013:TEXT:EN:HTML&src=0), due to the fact that it is majority owned by the Swedish state. + +####Contracts - still a soild source for single stories +Journalists can use contract data to tell the stories on questionable [recipients of contracts](http://euobserver.com/institutional/116152) or [no-bid contracts](http://en.wikipedia.org/wiki/No-bid_contract), when authorities "forgets" to announce a formal tender process. For many stories journalists have howeever been deemed to [rely on reports from readers](http://www.guardian.co.uk/society/2012/mar/15/nhs-services-tender). An open and searchable database of contracts would innevetable improve the possibilities for covering such procurement processes more systematically. Could it for instance be possible to create an alert, which notified journalists whenever a payment was executed to a supplier (from our spending data), while not appearing in the register for official contract winners (from TED)? + +Looking outside the realm of journalism it is worth noting that transparency organisations and international institutions recently have begun looking at more closely at contract data, though until now mostly outside the EU. The last year has seen several new initiatives on procurement including [Opencontracting](http://www.open-contracting.org/) of the World Bank Institute and another [procurement initiative](http://sunlightfoundation.com/blog/2013/03/11/announcing-a-new-procurement-initiative/) in the making from the Sunlight Foundation. A few weeks back the folks at WBI helped lay out some ground work for [a draft for a data standard](https://github.com/birdsarah/oc-datamerge-spike/tree/master/draft-standard) based on procurement data from the UK, US, Columbia, the Phillipines and the World Bank. For EU procurement data, I've helped write up a similar suggested draft at [OpenTed](http://opented.org/getthedata/data-standards-in-procurement/). +Tim Davies recently wrote up this account on the work to sort out a [draft for a contracting data standard](http://www.timdavies.org.uk/2013/04/04/developing-data-standards-for-open-contracting/). Initiatives such as these could hopefully be helpful for opening up EU procurement data to journalists. + +So join at the Procurement Hack Day in Brussels on [May 2](http://procurement-hack.eventbrite.com/#) as well as at the [Dataharvest](http://www.journalismfund.eu/dataharvest13). diff --git a/examples/openspending/content/blog/2013-04-22-Opening-the-Budget-of-Israel.mdown b/examples/openspending/content/blog/2013-04-22-Opening-the-Budget-of-Israel.mdown new file mode 100644 index 00000000..dd46dc04 --- /dev/null +++ b/examples/openspending/content/blog/2013-04-22-Opening-the-Budget-of-Israel.mdown @@ -0,0 +1,56 @@ +--- +authors: +- anders +redirect_from: /2013/04/Opening-the-Budget-of-Israel/ +title: Opening Israel's Budget +--- + +Public Knowledge Workshop + + +This is a guest post by Aviv Sharon and Adam Kariv from [The Public Knowledge Workshop](http://www.hasadna.org.il/en/) based in Israel. +
    +"Show me the money!" If you ask the developers of "Taktziv Patuakh" (Hebrew for "Open Budget"), one of the projects of Israeli non-profit “The Public Knowledge Workshop”, that's more than just a line from a film. The government's true priorities are reflected in the budget, not in its statements. That's what makes budgets interesting. And that’s why looking at budgets is one of the most important things that the public should be demanding from the government. As the negotiations for Israel's 2013 budget draw near, informed public discourse on budgetary matters becomes an even more pressing need. + +There are a few important decision-making points in the budgeting process that are most important for the Israeli public to keep track of. First, each law and ordinance gets an overall, macro budget. Then, that budget is broken down into smaller items, on which the state can spend money for purposes like paying salaries, ordering equipment, supporting various non-profits and more. If we, as the public, don't keep track of the whole process, the state can deceive us in several ways: The state might decide on a policy but not budget it, it might budget it but not spend the cash, and it might spend it in ways worth watching closely, like paying suppliers that were chosen without conducting a tender. In short, there are too many leaky holes in this pipe, and it's hard for the public to follow the money. + +The flow of money through Israel's coffers became a concern for the developers of "Open Budget" following a large [forest fire on Mount Carmel in December 2010](http://en.wikipedia.org/wiki/Mount_Carmel_forest_fire_%282010%29), the deadliest and most widespread fire Israel had ever seen. After the fact, Israel's ministries of Finance and Interior blamed each other for having neglected the fire brigades. On the one hand, the Ministry of Interior claimed that its requests to increase the budget for the brigades were only partially funded by the Ministry of Finance, and very late at that. On the other hand, the Ministry of Finance claimed that it had passed along all the sums stipulated in the relevant governmental resolutions. The Ministry of Finance also added that these resolutions had even greatly increased the funds for aerial firefighting in 2010 compared with previous years. + +###Open Budget 1.0 +To clear up the argument, activists from Israeli non-profit [The Public Knowledge Workshop](http://www.hasadna.org.il/en/) asked Michael Eitan, then Minister of Improvement of Government Services, to help retrieve data they could use to put together a comprehensive picture of Israel's state budget. Unfortunately, the Israeli Ministry of Finance did just the bare minimum and released several files in various formats, such as Excel, PDF and html. The files were rife with human errors, like parts of the Hebrew text mistakenly written left-to-right. (Hebrew is written from right to left.) + +First, the "Open Budget" developers at the Public Knowledge Workshop took the data and organized it in a uniform database. Secondly they built a web-based interface allowing any user to explore the budget. Eventually the database of the Public Knowledge Workshop reached a stage where this was the most accurate and comprehensive database of Israel's budget. Some ministries have even asked to export data from the Open Budget for them, because the state systems were much more cumbersome. + +The first version of [Open Budget 1.0](http://open-budget.readthedocs.org/en/latest/) was launched in March 2011 and is still available on [Israel's governmental services portal](http://budget.msh.gov.il/). It allows the public to track budget changes over time, search items for particular keywords and compare the budgeted amounts with the sums spent on each item.
    + +###Open Budget 2.0 +Following the first release, the "Open Budget"-developers realized that many of their basic assumptions for analyzing the data were wrong, and that several important features were missing in version 1.0. + +For instance, they knew that each item of the budget had an ID number, like item 15 for the Ministry of Defense budget, and assumed that such ID numbers would be consistant over time. This turned out however to be wrong. The Ministry of Finance changed the meanings of many items and sub-items from year to year, and therefore tracking spending on a given item over time based on the ID number seemed meaningless. While most high-level numbers usually kept similar meanings over time, the Ministry of Education (Item 20), for example, had changed responsibilities over time: Namely, it was in charge of sports and culture in some years, but not in others. This made it hard to track the education budget properly over time. Regrettably, the IDs on lower level budgetary items were detected to be even less stable in meaning, and little or no annotations were made available from the government to help observers discern the semantic changes. + +Moreover, to track spending on issues such as for instance firefighting, one would need to put together the sums spent on various different items, paying attention to the differences between salaries, procurement and other items. +Last but not least, while the budget may contain all the data, one can't easily determine if the sums are too much, too little, or just right. Nor can that be determined objectively. This is a value judgment, and different people will analyze the budget in different, and sometimes contradicting, ways, and use different data to frame and support their arguments. For example, one could argue spending in the Ministry of Education was appropriate 30 years ago, but has not kept up with the number of students and inflation. Hence, a good system for viewing the budget would allow one to easily examine spending versus those data. What’s more, a good system would then allow experts and civic activists in Israel to publicly annotate budgetary items with their respective opinions, to enrich the budgetary debate with accessible, evidence-based arguments. + +These needs are addressed in Open Budget 2.0, which is due for release from the Public Knowledge Workshop within the next few weeks. Namely, the next version of the system will allow users to:
    +- unify related budgetary items,
    +- analyze them together,
    +- compare spending between different items, or between budgetary items and other data sets, which can help put the budget in a larger context, and
    +- publicly annotate budgetary items.
    + +###Looking Ahead +Our dream at the Public Knowledge Workshop is to make any Israeli governmental expense as transparent as reasonably possible, in real time. + +Predictably, one of the biggest roadblocks for progress on budgetary transparency comes from the government itself. Currently we are trying to make the most of the data, which have already been released, but the Ministry of Finance and other branches of government still hold a lot of data sets which they reject to share. Open Budget developers suspect that until they are forced to release them, it will stay that way. Some examples of data sets yet not released are:
    +- state-provided evaluations of the effect of each budget item on the economy,
    +- budgets on the ministerial level,
    +- budgets of governmental agencies,
    +- details of individual procurements, and
    +- much greater detail on outsourced activities.
    + +###Recent changes +Presumably as part of the changes made by the new Israeli Minister of Finance, Yair Lapid, the [2012 budget was made fully available](http://data.gov.il/dataset/534) to the public on April 12, 2013. Amazingly, a representative of the Ministry personally contacted the Public Knowledge Workshop to verify that activists at the Workshop had seen the new release. Hopefully, this is a sign of more openness to come. + +We encourage new volunteers to [join the effort](www.hasadna.org.il). There are many important causes to contribute to, but extracting knowledge from a bunch of numbers, that is a real challenge. +
    +Aviv Sharon is a volunteer at the Israeli non-profit "The Open Knowledge Workshop". He writes materials for the public and plans educational projects.
    +Adam Kariv is a board member of "The Open Knowledge Workshop" and project leader of "Taktziv Patuakh" (Hebrew for "Open Budget"). diff --git a/examples/openspending/content/blog/2013-05-14-Introducing-the-Progress-Page.mdown b/examples/openspending/content/blog/2013-05-14-Introducing-the-Progress-Page.mdown new file mode 100644 index 00000000..f3d01118 --- /dev/null +++ b/examples/openspending/content/blog/2013-05-14-Introducing-the-Progress-Page.mdown @@ -0,0 +1,26 @@ +--- +redirect_from: /2013/05/Introducing-the-Progress-Page/ +title: Introducing the Progress page +--- + + + +For the past months we've seen many new users come to the OpenSpending community. The community have now added budget data from more than [80 cities](http://apps.openspending.org/maps/) and most recently [universities](http://openspending.org/ugr-spending) have also found their way onto OpenSpending. + +While we're thrilled by the uptake in use of openspending for visualising budgets, we've however also noticed that datasets from transactional spending are added less often than we'd hope for. We would therefore want to make it easier for the community to track what other people are working on and offer their help if needed in sutiations like these:
    +- "I'm working on a dataset, but got stuck cleaning the spreadsheet - who can help?"
    +- "I've found this excellent dataset on Italian structural Fund payments and would like to add it to OS, but need help translating the columns before uploading."
    +- "I wonder if someone is already working to add the latest EU budget to OpenSpending?"
    + +### Enter - the [Progress page](http://progress.openspending.org/) +With the Progress page we wish to make it easy to show what datasets people are working on. + +### What can you do there? +- Claim a dataset you'd like help to clean and upload to OpenSpending - and ask the list for help if needed +- Check what datasets other people are working on - and offer your help if you can +- Request a dataset you would like to see on OpenSpending + +At this early stage we're really interested to hear feedback from you, so let us know how we can make the Progress page useful to track how datasets are moving forward. Share your thoughts on our [mailing list](http://lists.okfn.org/mailman/listinfo/openspending). + +Do you feel like contributing to the code of the Progress page, check our issues on [Github](https://github.com/openspending/dataprogress/issues). + diff --git a/examples/openspending/content/blog/2013-05-27-Procurement-Hac-Recap.mdown b/examples/openspending/content/blog/2013-05-27-Procurement-Hac-Recap.mdown new file mode 100644 index 00000000..f02ae2ab --- /dev/null +++ b/examples/openspending/content/blog/2013-05-27-Procurement-Hac-Recap.mdown @@ -0,0 +1,53 @@ +--- +redirect_from: /2013/05/Procurement-Hac-Recap/ +title: Opening the EU procurement database +--- + +OpenTed datastore + +On 2 May civic coders and journalists from across Europe from Norway to Slovenia met up in Brussels for a Hack Day supported by [Knight Mozilla Open News](www.mozillaopennews.org/) to dig into the EU procurement register, [Tender Electronic Daily](ted.europa.eu). Here are a few of the highlights from the day and where we've been able to take the TED data since the Hack Day. + +### Scraping and parsing of TED into OpenTED +Several people got together to build and improve the parsing of data already scraped. The past years has seen several attempts to scrape and parse TED, but past scraping projects have been missing the granular parsing needed in order to make the data useful for investigations. + +You can find the latest updates here - we'd love your help: +https://github.com/opented/opented + +### User stories - ideas for data journalism on procurement data +During the Hack Day a handful of data journalists sat down to line up questions, which should be investigated by journalists. "If the TED data could speak, what should we ask?" + +####Key questions to ask the TED data: +- Where in the EU are highways most expensive to build? +- What are the best ways to expose cartel formations +- What companies are running food services at public school (and what's on the menu)? +- Who is the biggest weapons salesman in the EU? +- Who are the biggest public transport providers - Arriva vs. Connexion? +- What's the relationship between procurement process (time) and what companies gets the contract? +- How many contracts go outside the EU? +- Do the governments purchase from big or small suppliers? +- How can we predict when a project will go over budget? +- What sector is the least competitive (CPV-code vs. average bid)? +- Are there companies within certain sectors that get a suspicous large amount of contracts? + +####Using other data with procurement data from TED +- Reconcile with company data from Opencorporates +- Match contracts to actual spending data eg. [UK government spending](http://openspending.org/ukgov-25k-spending) +- Ask Freedom of Information requests to the procuring authority for entire contract (see examples of such successful FOI requests [here](http://www.asktheeu.org/en/request/292/response/805/attach/2/Signed%20Framework%20Agreement%20with%20Eurocontrol.PDF.pdf) and [here](http://www.asktheeu.org/en/request/293/response/909/attach/3/answer%20del%20BD%20GESTDEM%202012%205786.pdf) +- Get to understand what the contracts are actually about with the [official taxonomy of the CPV-code](http://simap.europa.eu/codes-and-nomenclatures/codes-cpv/codes-cpv_en.htm) + +### A first look at the TED data +The [full TED data set](http://opented.pudo.org) is now available, thanks to the hack team and in particular [Friedrich Lindenberg](twitter.com/pudo). +You can access the TED data as CSV files EU wide or on a country by country basis. You can also get to know the data format without downloading it, as we've uploaded the [data for Czech Republic (2013)](https://docs.google.com/a/okfn.org/spreadsheet/ccc?key=0AvdkMlz2NopEdEtxU3dDYktRT3ltbTdadnZBcmdaVVE#gid=0) as a sample. + +We hope that many of you will dig into it, and examine the contracts. + +####Reviewing the data quality +At a community call on May 15, community members decided to begin a quality review of the data available in TED. +We're asking you to help contributing to this [review](https://docs.google.com/a/okfn.org/document/d/16tXDgtMQVIX5TnzWPPZScRZEd0FkW14Zcc_lche5T_w/edit#heading=h.t1acmd7rub4n). If you'd like to help, please get in touch! In particular the review will examine missing amounts (non mandatory fields), missing contract fields and inconsistent data formats. The quality review can be really useful in order to get a clear picture of the state of the TED-data and help assess how useful it will be for journalists and researchers. We'd be particular interested to hear from those of you who are already working on procurement data in EU countries. +Thanks to [Julia Keseru](https://twitter.com/jkeserue) from Sunlight Foundation and [Ronny Patz](https://twitter.com/ronpatz) Transparency International, Brussels for offering input on this. + +###Should OpenSpending include procurement data? +Procurement data is increasingly becoming available in more countries such as [Senegal](http://openspending.org/marches-publics-senegal/entries), [Ukraine](z.texty.org.ua) and [Australia](http://openspending.org/australian_federal_government_contract_spending), where transactional spending data remains unavailble. By definition procurements are clearly different from transactional spending data. While "contract awards" should indicate a final price, they might not be final, as ammendments regularly occur in contracts and did anyone ever hear of an ICT-project, which went over budget? In such a case chances are that you as a journalist or watchdog would prefer to have access to the actual spending data, but if this is not available procurement data can still help you get closer to your story. Contracts might also lack clarity as to when payments are to actually occur. Finally procurement data will most often only include contracts above certain thresholds, which makes significant shares of actual contractor spending unaccessible. +
    I could be relevant for OpenSpending to add procurement data next to bugdet and spending data, because several countries with available procurement data, do not provide access to the ideal option of transactional spending data. This is for instance the case in most EU member states where procurements from OpenTED today is likely offering the most granular data on public spending. + +Do you think OpenSpending should begin adding more procurement data? Let us know! diff --git a/examples/openspending/content/blog/2013-06-02-french-press-aid.md b/examples/openspending/content/blog/2013-06-02-french-press-aid.md new file mode 100644 index 00000000..ca420fdf --- /dev/null +++ b/examples/openspending/content/blog/2013-06-02-french-press-aid.md @@ -0,0 +1,22 @@ +--- +redirect_from: /2013/06/french-press-aid/ +title: Making sense of French press aid +--- + +This is a guest post by [Samuel Goëta](https://twitter.com/samgoeta), OKF France. + +On May 22nd members of the [OKFN local group France](http://fr.okfn.org/) met at la Cantine, the digital hub of Paris, for visualising press aid data. This workshop held prior to a conference organised by the Social Media Club France on the efficiency on press aid for innovation in the media industry. The results of the 3-hour intensive workshop were presented right after to the audience and the speakers of the conference. + +The press has received support in France since Second World War as it is considered critical for ensuring all opinions can circulate in society. These aids are direct subsidies as well as indirect in the form of tax breaks, reduced post fares and subsidised distribution. More recently, in April 2012, the press industry was granted the Strategic Found for the Development of the Press (fonds stratégique pour le développement de la presse) accounting for 60 millions euros of aid. This found was associated with a commitment to transparency, data on the 9 million euros spending, was published in April 2013 on the French [open data portal](http://data.gouv.fr). + +photo (3) + +Following the publication of data on press aid, 5 members of OKF France got hands on the data available. Every participants had no previous experience with data visualization and did not even prepare this workshop. The challenge was to prove that, using open resources and with perseverance, one can visualise a complex system of subsidies in a couple of hours. Data cleansing was obviously the hardest part of the work as participants had little experience of handling data. + +The workshop was divided in two groups, one using Open Spending to visualise data on an interactive treemap. + + + +Another group used D3 Javascript library to map data on [an interactive bubble chart](http://okf.s3-website-eu-west-1.amazonaws.com/aidesalapresse/). + +The presentation of the visualizations between the two roundtables of the conference raised great interests, as most speakers were speaking about data they could not visualize nor source. The results were directly [published on checkthis](http://checkthis.com/y2ci). The poster spread quite quickly as it reached 400 views in less than 24 hours. This workshop shows that mediation is crucial if we want to develop data literacy. Open data publishers should encourage and support such events as they can also benefit from simple visualisation everyone can use and understand. It also shows the limits of transparency on press aid: although aid is distributed for the development of innovation projects, the details of the funded projects are not mentioned. Officials state it could not be published based for trade secrets reasons. Also we were able to visualize only the tip of the iceberg as data is published for 30 millions € of press aid while total of direct and indirect aids accounts for 1,2 billion € in 2012. This workshop was a small, probably incomplete, effort to understand a complex and opaque funding system for sustaining an industry in the midst of massive changes. diff --git a/examples/openspending/content/blog/2013-06-04-dc-spending-data.mdown b/examples/openspending/content/blog/2013-06-04-dc-spending-data.mdown new file mode 100644 index 00000000..8cd2efce --- /dev/null +++ b/examples/openspending/content/blog/2013-06-04-dc-spending-data.mdown @@ -0,0 +1,17 @@ +--- +authors: +- anders +redirect_from: /2013/06/dc-spending-data/ +title: Hack for Change helps publish data from Washington DC +--- + + + +
    +This weekend thousands of people gathered across United States for a [national day of civic hacking](hackforchange.org). OpenSpending joined organisers at [Code for DC](http://codefordc.org/) in Washington to work on transactional spending data published by the city for 2010 to 2013. The data now allows you to browse spending on [charter schools](http://openspending.org/dc-vendors-contractors/from/public-charter-schools/entries#from:public-charter-schools) and contractors in the [corrections industry](http://openspending.org/dc-vendors-contractors/from/department-of-corrections/entries#from:department-of-corrections). + +During Hack for Change we met with journalists and citizens to hear more about what kind of questions that could be answered with the data available. + +If you're interested to learn how you can get spending data from your city onto OpenSpending, get in touch via our [mailing list](http://lists.okfn.org/mailman/listinfo/openspending) or [Twiiter](https://twitter.com/openspending). + + diff --git a/examples/openspending/content/blog/2013-06-05-New-Features-in-May-2013.md b/examples/openspending/content/blog/2013-06-05-New-Features-in-May-2013.md new file mode 100644 index 00000000..3d8470c0 --- /dev/null +++ b/examples/openspending/content/blog/2013-06-05-New-Features-in-May-2013.md @@ -0,0 +1,91 @@ +--- +authors: +- tryggvib +redirect_from: /2013/06/New-Features-in-May-2013/ +title: New Features in May 2013 +--- + +![Image by Mosman Library (cc-by 2.0)](https://farm9.staticflickr.com/8205/8206356927_3d5c720658_c.jpg "Badges!") + +This is perhaps not a very attention grabbing title for this post, but some cool new features have landed in OpenSpending this month. We're really proud of them so we thought we'd share them with you. + +### Badges + +We've got badges! OpenSpending administrators can now award badges to outstanding datasets. To begin with there are only a few badges but we foresee collaboration with organisations that want to give their approval to datasets. These organisations can do quality assurance on datasets originating from them but have been modified to fit better into OpenSpending (the badge would then say: *"Yes this is still the data we published"*). These organisations could also give badges to datasets that help their cause (the badge would then say: *"This dataset helps us reach our goal"*). + +If you're representing an organisation and want to be able to give out badges, please [get in touch](http://openspending.org/about/contact.html). + +If you're managing a dataset, find out which [badges you can get](http://openspending.org/badges) and start collecting! + +### Show Analysis Results + +After loading a source users can't really see if the load was successful. It may appear to be successful but that might be just because OpenSpending was able to download something. + +To provide some kind of feedback OpenSpending now shows the results of the analysis it does on the data. In particular it shows the columns that it found. If there aren't any columns or the column names are weird, users might now catch it before something goes wrong. + +### EU Cookie Compliance + +This may not be a big thing for you and it might even be slightly irritating but for us this is really important. We want to follow the law and OpenSpending now does so by implementing the EU Cookie Directive (if you want to know about cookies being placed on your computer this is important to you too). + +Users are now presented with a small banner that tells them about the cookies and offers them link to a page where they can read more about [our cookie policy](http://okfn.org/cookie-policy/). + +### OpenSpending Contacts Map + +There are many projects out in the world that work in some way with spending data. We want to be able to connect those initiatives together. They can attract new contributors, learn from other projects and spark new interesting projects (even in other countries). + +To help you establish these connections or to find projects you are interested in we've put up an OpenSpending app on [apps.openspending.org/oscontactsmap](http://apps.openspending.org/oscontactsmap/). There you can see the world and find all of the projects that relate to OpenSpending in some way. + +If your project isn't there, don't panic! Thera are lot of projects so we might have missed some of you. Please let us know about your project and we'll add it to the list right away. When we're sure we have most of the projects on the list we'll make this map more prominent on OpenSpending's main site. + +### Satellite Template + +Before May users had to fork [Where Does My Money Go?](http://wheredoesmymoneygo.org/) or some other site in order to create their own satellite site (a site that provides context to the data and analysis in OpenSpending). + +In May we created a satellite site template so you can easily recreate sites like *Where Does My Money Go?* with a simple config file without having to remove a lot of context specific information. + +Just fork [our satellite template repository](https://github.com/openspending/satellite-template) and start configuring your own satellite site. + +### Other Changes + +There are loads of other smaller changes we did this month. We went through all of the issues in our issue tracker that were labelled as *bug* and fixed them. We tried to make some instructions clearer. We made small headway in getting better IE7 compatibility and we now show the type of a dimension that has been created with the model editor. + +### Want to contribute? We need code reviewers! + +We're really proud of what we've achieved this month. There are a lot of upcoming changes in the pipelines (in the form of pull requests). Quality assurance is really important to OpenSpending so we don't add anything new to the platform unless it's been looked at by at least one other developer (this is called code review). + +If you're interested in contributing to OpenSpending, code reviews are a good place to start. You get familiar with the code base and you can (and should) raise all kinds of questions (so if in doubt about anything, just ask the developers). + +There's even a law in open source software (Linus' Law): *"Given enough eyeballs, all bugs are shallow"*. That means the more people do code review, the better OpenSpending will get. That's why we'd appreciate your help in getting some more quality code into OpenSpending. Look at the pull requests in our repositories on GitHub and comment, both if you see something fishy and if you don't! + +**Interested in being a code reviewer? [Get in touch!](http://openspending.org/about/contact.html)** + +###Data Clinics and Community Hangouts - going forward +A lot of great spending projects in the making are in the need of community support for tips on how to clean and upload the data. + +For this reason we'll therefore begin to run bi-weekly Data Clinics - a community drop in for all kinds of technical troubleshooting you need in order to get working on OpenSpending. +All you need to bring is your messy data, and you'll be able to ask the community on tips on how to tackle your spending data. + +The first Data Clinic will take place **today Wednesday 5 June at 19:00 CET / 18:00 BST / 13:00 EDT** as chat via skype (ping: anpehej) or IRC #openspending. [Sign up here!](https://docs.google.com/a/okfn.org/document/d/1vx9oSJieuqfQ2xdBTyFSTmDRsPRenTrJLKJDo_ZP8z4/edit#) + +With the introduction of the Data Clinic we'll now offering community activities every Wednesday - either as Data Clinic or Community GoogleHangout. We hope that you'll join some of the activities! + +###Current community projects +Here are some of the projects currently in the works. Many of them could need a helping hand: +- Felix is working on Andalusian procurement data. Data quality seems good, and would be interesting to compare with the data available in OpenTED. +- Cecilia and the investigative journalism team at IRPI has claimed the Italian structural fund data and will be working to add it to OpenSpending. They need help cleaning a few of the data fields so please get in touch if you can help! +- Tarek would be interested looking at spending data from Egypt +- Pierre has helped scrape, clean and upload procurement data from Senegal to OpenSpending. He'll post on the OpenSpending blog soon. +- Benjamin is working on the EU procurement data opendted.pudo.org +- Transactional spending data from the city of [Washington DC](http://openspending.org/dc-vendors-contractors) was added during theweekend as part of #Hackforchange. + +###Global Weekend on city spending data July 20-21 +The past months has seen strong growth in local spending projects with [86 cities mapped](http://apps.openspending.org/maps/). As we want to support and grow these initiatives we're asking if you'll join us to organise a global spending weekend to open city spending on July 20-21. The spending weekend will offer online trainings and support to your local group to help you get your project off the ground. +Hiroichi (OKF Japan), Mor (Hasadna) and Adam (Open Budget Oakland) have already expressed interest in the idea, and we'd love to hear if you want to be involved. + +Have a look and add your self to the [planning document](https://docs.google.com/a/okfn.org/document/d/1Zh-TPxgMiFDrzk-rNJqL9CmCbbtlZmp2xjWlZ6T20TA/edit#heading=h.6416s7qbnyrt). + +### Thanks + +Thanks to **Nigel Babu**, **Lucy Chambers**, **Martin Keegan**, **Anders Pedersen**, **Rufus Pollock**, **Stephen Russett**, and **Stefan Wehrmeyer** for their contributions this month (there are probably a lot more who've contributed somehow to this month's features so don't be sad if we forgot you, let us know and we'll add you). + +Image of badges used for this blog posts is by [Mosman Library](https://www.flickr.com/photos/mosmanlibrary/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). diff --git a/examples/openspending/content/blog/2013-06-05-amsterdam-citizen-budget.md b/examples/openspending/content/blog/2013-06-05-amsterdam-citizen-budget.md new file mode 100644 index 00000000..3c45f179 --- /dev/null +++ b/examples/openspending/content/blog/2013-06-05-amsterdam-citizen-budget.md @@ -0,0 +1,34 @@ +--- +redirect_from: /2013/06/amsterdam-citizen-budget/ +title: Budget monitoring in an Amsterdam neighborhood +--- + +Budget monitoring in an Amsterdam neighborhood +The Centre for Budget Monitoring and Citizen Participation (www.budgetmonitoring.nl) is a Dutch organization founded in December 2011. The foundation is the result of a co-operation between active citizens and social workers from the Netherlands and INESC from Brazil (www.inesc.org.br), supported by Oxfam Novib through E-Motive. +The Centre is rooted in the Dutch social movement and implements budget monitoring as a tool to enable citizen’s access to financial information, to promote social participation in public policy making and to monitor the spending of determined budgets of various governmental organizations. In this way, the centre attempts to strengthen civil society and enhance social participation. +Indische Buurt +In 2012 we started with an experiment in one of the neighborhoods in Amsterdam, called Indische Buurt. The Indische Buurt was chosen for the experiment on budget monitoring for various reasons. For example, many active citizen groups, called communities exist and are active within the neighbourhood. These groups already work hard to improve their neighbourhood and are actively developing numerous instruments on participation. Budget monitoring suits their approach well. +Besides, these communities take on a very specific starting point in their work; namely, the citizen and his right on ambition, which can be fulfilled in several ways in his neighbourhood. The communities play an important role to fulfil these ambitions. Budget monitoring and participation contribute to the right of ambitions. +And finally, budget monitoring cannot be implemented as an instrument on its own: it has to be rooted in social movement. The communities in the Indische buurt were already active within various areas therefore, it has been convenient to experiment with budget monitoring in collaboration with these active groups. + +To start the experiment on budget monitoring based on the methodology of INESC it was necessary to translate the methodology for the Netherlands into a roadmap. The university of Applied Sciences of Amsterdam developed the roadmap Budget Monitoring based on the methodology of INESC , and adapted it to the Dutch situation, after discussions with the communities in the Indische Buurt. In the roadmap, the emphasis is placed on social justice and civic participation and less on human rights in comparison to the methodology of INESC. Budget monitoring comprises of five steps. + +The steps towards budget monitoring +In the first step we tried to localize public budgets. This was not very easy because budgets and information about spending are not (yet) transparent in the Netherlands. So, we had to search for information in PDF-documents. In Amsterdam the annual reports and budgets, as well as ‘5 and 9-months reports’ and reports of meetings have to be published at www.oost.amsterdam.nl (and www.amsterdam.nl). It is very difficult to find the proper information, so an expert was needed to find the way in the 1000 pages on the sites. After a lot of research, we built a simple database. This database also included projects in the city of Amsterdam, concentrating on neighborhoods. +In the second step we analyzed the database to understand the budget for the Indische Buurt. We were unable to find all the budgets that were spent in the neighborhood. This turned out to be a common problem with civil servants. As a civil servant told us in January 2012: “We don’t know the budget for the Indische Buurt, every civil servant just knows his own budget”. +This heavy conclusion was also the reason for the local district to start a research on this subject. The result was a pilot on neighborhood budgets. In the next blog we will write more about this. +During the third step we organized a training. The participants of the training were spokespersons of communities and other community members. The subjects matters were: budget cycle, annual report, annual budget. And other subjects, such as how to influence politics, how to make a plan for the neighbourhood? + +Part of the training was the practice and theory of budget monitoring in Brazil, by trainers of INESC who showed the group the emphasis on political influence and advocacy. During the training we started to analyze the data available on budgets. For example, we compared the budget of the local district 2011 with the budget of 2013. We also studied a list of subsidies from the city of Amsterdam and the Ministery of internal affairs for the Indische Buurt. + +In the meantime, we asked the local district to give us more information. The district, also began to conduct research. The first result of our request was a short version of the annual report 2011 with infographics. +To make a proper Agenda the group decided that they had to know the priorities of the citizens. What do citizens want for their neighborhood? A questionnaire was made based on the results of several participation events in the Indische Buurt. + +After detecting the priorities of the citizens, we re-monitored the budget of the local district and re-analyzed the annual report of 2011. We did this by comparing the figures, the local district published. And we noticed, for example, that there was a big difference in 2011 between the budget and the spending of the budget on education, youth, and welfare. Altogether the difference was over 3,2 million euros. + +On the basis of the findings during the training and questionnaire, the community members decided to make a perspective paper, with a long-term policy, instead of making just an Agenda for the neighbourhood. The training group used the financial data and went to the political board of the local district to ask questions about budgeting and spending. +The report of this experiment and the questions the communities asked can be seen at: http://www.budgetmonitoring.nl/english/downloads/Brochure%20Budgetmonitoring%20(ENG).pdf. +The report also contains the way INESC is working on budgets from the perspective of human rights. + +The experiment has now concluded. But, the communities and the Centre will keep working with budget monitoring. In the past 3 months things changed a lot. For example, the local district made a website containing facts and figures per neighbourhood. In our next blog we will pay attention on this subject. + diff --git a/examples/openspending/content/blog/2013-06-10-Next-Community-Hangout.md b/examples/openspending/content/blog/2013-06-10-Next-Community-Hangout.md new file mode 100644 index 00000000..74827739 --- /dev/null +++ b/examples/openspending/content/blog/2013-06-10-Next-Community-Hangout.md @@ -0,0 +1,23 @@ +--- +authors: +- anders +redirect_from: /2013/06/Next-Community-Hangout/ +title: Next Community Hangout 12 June +--- + +We'll be running another Community call via Google Hangout Wednesday this week. + +**When:** 12 June, 19:00 CET / 18:00 GMT / 13:00 EDT - [your time zone](http://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenSpending+Community+Hangout&iso=20130612T13&p1=263&ah=1) +
    +**Where:** On Google Hangout. Sign up in the form below. +
    +**Agenda:** +
    +- “How to crunch numbers from the European Investment Bank (EIB)?”. Introduction by Anne Roggenbuck from [Bankwatch](http://bankwatch.org/) about how they analyse loan data from the EIB.
    +- News from around the community: Who's got something to share - who needs help moving forward?
    +- Procurement data on OpenSpending: French and Senegalese procurement are among the most recent contributions to OpenSpending. Should this lead to any changes in the way [the upload module is set up](http://openspending.org/datasets/new)?
    +- Planning the [Global Spending Weekend July 20-21](https://docs.google.com/a/okfn.org/document/d/1Zh-TPxgMiFDrzk-rNJqL9CmCbbtlZmp2xjWlZ6T20TA/edit#)
    +
  • Updating budget and spending data in the [Open Data Census](http://okfnlabs.org/opendatacensus/country/). Is your country up to date? +
    + + diff --git a/examples/openspending/content/blog/2013-06-12-How-to-understand-data-from-EIB.md b/examples/openspending/content/blog/2013-06-12-How-to-understand-data-from-EIB.md new file mode 100644 index 00000000..7b2087fb --- /dev/null +++ b/examples/openspending/content/blog/2013-06-12-How-to-understand-data-from-EIB.md @@ -0,0 +1,58 @@ +--- +authors: +- anders +redirect_from: /2013/06/How-to-understand-data-from-EIB/ +title: How to understand data from the EIB - what does the data tell us? +--- + + + +This is a guest post by Anna Roggenbuck from [Bankwatch](http://bankwatch.org/). +
    +The European Investment Bank (EIB) is the house bank of the EU. It has a lending portfolio of about 60 billion euros annually, about 90 percent of which go to projects within the EU. + +For each project the EIB is considering to finance, it publishes on its website basic data including a description of the project, its location, the amount of EIB financing, in most cases the loan beneficiary and where relevant environmental impact assessments. The data has its limitations: no machine-readable data is readily available, and it is not possible to aggregate loan volumes. + +Machine-readable datasets for individual industry sectors are available on request, though not for all industry sectors (no transport database for instance) and you’ll have to wait one month for the file. With these files you can relatively quickly aggregate and examine EIB lending in a certain country or region like old or new Member States and for different years. Slightly more complex, but still very much possible, is collecting data for sub-sectors (e.g. renewable energy). This is particularly useful if comparing for instance energy efficiency lending in the old Member States of the western EU versus the new Member States, where the potential of and need for energy efficiency measures is much higher. + +EIB loans for energy efficiency projects 2007-2011:
    +- Old EU member states (EU 15): EUR 1,429 million
    +- New EU member states (EU 12): EUR 354 million
    + +At the same time, the data provided by the EIB is relatively sparse, and the files notably do not include the names of loan beneficiaries, which would allow assessing whether any companies receive particularly extensive EIB support and in which countries these are based. + +Another problem is how the EIB categorises projects. Even though the bank uses an official Eurostat classification system (NACE), inconsistencies and unclear attributions create a much rosier picture about the bank’s lending. (See more details for the energy sector lending below.) Note that for a more critical assessment or investigation one needs to consult additional materials like project summaries, EIB project assessment reports (only available on request), or a project promoter’s website. + +###Why is it important to analyse the EIB’s classifications? + +Take the EIB energy lending as an example. The EIB boldly states that it is a leading institution in the fight against climate change. While the bank does lend significant amounts for renewable energy projects, it also finances a large number of climate-damaging projects. + +What the EIB categorises as clean energy is often not clear. This is especially the case when portions of a particular loan are attributed to different, often conflicting EU priorities like financing for renewable energy and energy efficiency on the one hand and security of energy supply (to EU countries) and trans-European energy networks like gas or oil pipelines on the other hand. + +By assessing each project individually, we found the following underlying trends in the EIB’s overstated green energy lending: + +###1. Unclear classifications + +Quite a number of projects in the EIB database we simply removed because they did not seem to be related to energy, let alone renewable energy. For example the [Poland Forestry and Environment project](http://www.eib.org/projects/pipeline/2011/20110329.htm) aims at rural development and includes “afforestation, improved forest management and protection, and small-scale rural and agricultural infrastructure”. The project is classified as renewable energy in the EIB’s database, but after examining all the information we could find, it is inexplicable how the programme relates to renewable energy development. + +Confronted with our criticism of its classifications, the EIB argued that Bankwatch analysis is misleading as it lowers EIB’s support to renewable energy (by not considering large hydro and waste to energy projects) and to energy efficiency projects (by neglecting efficiency gains of new co-generation projects). Bankwatch explained that its methodology focuses on sustainability objective (what was an accusation from the EIB side) thus it takes into account wider impacts of projects, like biodiversity impacts of big dams or climate impact of new, greenfield fossil fuel power stations (even if they are considered more efficient than old ones, they are still new fossil fuel plants). EIB accused Banwkatch methodology of not being in line with EU policy (methodology) however Bankwatch had made its methodology transparent and explained main differences and justification for the methodology chosen. + +###2. Very broad strokes + +In some cases, financing that clearly did not go to renewable energy sources was nonetheless added to the renewables figure, even though renewables was just part of that loan. In case of [a project with the Italian power company ENEL](http://www.eib.org/projects/pipeline/2006/20060248.htm) the EIB assigned the loan to “Renewable energy” even though more than 50% of the total loan of 450 million Euro was aiming at fossil fuels. (See the excerpt from the EIB’s database below. Irrelevant columns were deleted to reduce space.) + +eib table +
    +###3. Compartmentalisation + +Somewhat in contrast to point two, the EIB in many cases focuses too narrowly on the specific activity they finance rather than taking the impact of a project as a whole into account. + +For example, a loan over 500 million euros in 2007 for the construction of [a new boiler at the Power Plant Karlsruhe](http://www.eib.org/projects/pipeline/2006/20060303.htm) is considered energy efficiency (and thus boosts the bank’s ‘green energy lending’) even though the project leads to an overall increase in CO2 emissions. + +This is especially problematic in the case of greenfield power plants in developing or transition countries that often have a significant potential to increase energy efficiency. EIB money that could be used for measures to decrease energy loss and consumption, has instead financed new and more efficient (or ‘less inefficient’) fossil fuel fired power plants to address growing energy demand. + +###What further questions can we ask? + +While the EIB's 2011 energy figures have improved over previous years, the share of fossil fuels is variable when considered over a longer period of time. It will be interesting to see whether the positive trend from 2011 continues. + +Up to 30 percent of EIB financing is offered as ‘global loans’ to intermediary institutions like commercial banks and different investment funds. Details for such activities are almost never available due to confidentiality provisions by the financial institution. A more in-depth investigation into these loans and the financial intermediaries could offer insightful information on how climate-friendly the EIB’s global loans are. diff --git a/examples/openspending/content/blog/2013-06-14-Data-Clinic.md b/examples/openspending/content/blog/2013-06-14-Data-Clinic.md new file mode 100644 index 00000000..9f0b4ccb --- /dev/null +++ b/examples/openspending/content/blog/2013-06-14-Data-Clinic.md @@ -0,0 +1,18 @@ +--- +authors: +- anders +redirect_from: /2013/06/Data-Clinic/ +title: Next Data Clinic 19 June +--- + +Earlier this month we organised the first OpenSpending Data Clinic, to offer help on how you can work with your budget or spending data. +At the first clinic we gave feedback on how to format the Canadian budget and how to convert PDFs from the budget of Barcelona. +
    +The next Data Clinic will be on Wednesday 19 June at 19:00 CET / 18:00 GMT / 13:00 EDT - [your time zone](http://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenSpending+Data+Clinic&iso=20130619T13&p1=263&ah=1) via Google Hangout and IRC channel #openspending. +
    +Bring your spending data and see if the community can help you out:
    +- How to visualise budget data?
    +- How to convert spending data locked in PDFs?
    +- How to format the data the right way?
    + + diff --git a/examples/openspending/content/blog/2013-06-17-universities-on-the-spending-map.md b/examples/openspending/content/blog/2013-06-17-universities-on-the-spending-map.md new file mode 100644 index 00000000..450e3178 --- /dev/null +++ b/examples/openspending/content/blog/2013-06-17-universities-on-the-spending-map.md @@ -0,0 +1,46 @@ +--- +authors: +- anders +redirect_from: /2013/06/universities-on-the-spending-map/ +title: Budget transparency for an open university +--- + + + +This is a guest post by [José Félix Ontañón](https://twitter.com/fontanon) of [Openkratio](http://openkratio.org/). + +The project began by scraping budget documents in PDF format published by the University of Grenade to a machine-readable format with the purpose of making it easier for citizens, journalists and even employees at the university to work with the data, using tools from spreadsheets to visualization suites. The resulting app allow users to download the income and expenditure budgets in CSV format and provides a set of comprehensive visualizations. + +The idea emerged at the International Open Data Day, a gathering of citizens in cities around the world to write applications, using open public data to show support for and encourage the adoption open data policies by the world's local, regional and national governments. The Open Knowledge Foundation's local Spanish chapter did a call for participants and the Free Software Office at the University of Grenade accepted the challenge. + +The set of tools used for the project included: + +* [CometDocs](http://www.cometdocs.com/) (online pdf 2 excel converter) +* [Open Refine](http://openrefine.org/) (data cleaning) +* [DataHub](http://datahub.io) (data hosting) +* [OpenSpending API](http://docs.openspending.org/en/latest/model/design.html) & [D3.js](http://d3js.org) (visualizations) + +###The importance on University budget transparency + +The Spanish Public Universities are almost solely funded by several Public Administration Offices. With the University of Grenade as an example, the amount of income coming from Public Payments (including college tuition) only raise up to 11% of the total. As a result of the Spanish economic crisis some college tuition rates rose, having a deep impact on the +pocket of those on the verge of being able to pay their studies. + +By releasing University budgeting data, the project: + +* Highlighted the reality of the resources available at the university +* Helped identify potential best practices in savings that could be used by other universities +* Helped citizens to make smart proposals on why-and-where the public should invest in higher education + +Spanish public universities are equipped with system called SIIU (Integrated University Information System) and they are required to report budget data using this +system. Thus in reality most of the technical challenges around developing budgets in electronic and harmonized formats has already been completed. The question is therefore why the Ministry of Education will not make this information available to the public. + +###Reactions following the publication of the app +Honestly, budget transparency at universities is not a hot-topic today. Fortunately, a growing community in Spain where David Rey Jordan, a colleague at OpenKratio citizen group, member +of Open University group at OKFN Spain, and a public employee at Universidad Pablo de Olavide (Seville, Spain) encouraged me to replicate the same exercise for Olavide. Universidad Pablo de Olavide University opened its own Open Data Portal on 2012 releasing data about budgets and qualifications. Some visualizations using OpenSpending were done at the Open Data Day. + +Find Openkratio on [github](http://openkratio.github.io/ugr-presupuestos/) + +Nota: There are some weak points on the data. While income budget is broken down in +four levels, only three levels for the spending budget are available. The official University +spending budget document (with 5 levels) has been published as scanned PDF images +making them virtually imposible to be scrapped. diff --git a/examples/openspending/content/blog/2013-06-26-next-community-hangout.md b/examples/openspending/content/blog/2013-06-26-next-community-hangout.md new file mode 100644 index 00000000..fd056e1c --- /dev/null +++ b/examples/openspending/content/blog/2013-06-26-next-community-hangout.md @@ -0,0 +1,19 @@ +--- +authors: +- anders +redirect_from: /2013/06/next-community-hangout/ +title: Next Community Hangout 3 July +--- + +We'll be running the next Community Call via Google Hangout Wednesday 3 July. + +**When:** 3 July, 19:00 CET / 18:00 GMT / 13:00 EDT - [your time zone](http://www.timeanddate.com/worldclock/fixedtime.html?msg=OpenSpending+Community+Hangout&iso=20130703T13&p1=263&ah=1) +
    +**Where:** On Google Hangout. Sign up in the form below. +
    +**Agenda:**
    +- Taxation and open data: Follow up on discussion from mailing list and the [G8 meeting](http://blog.okfn.org/2013/06/25/what-data-needs-to-be-opened-up-to-tackle-tax-havens/)
    +- News from around the community: Who's got something to share - who needs help moving forward?
    +- Status on the [Global Spending Weekend July 20-21](https://docs.google.com/a/okfn.org/document/d/1Zh-TPxgMiFDrzk-rNJqL9CmCbbtlZmp2xjWlZ6T20TA/edit#)
    +
    + diff --git a/examples/openspending/content/blog/2013-06-30-spending-data-party-announce.md b/examples/openspending/content/blog/2013-06-30-spending-data-party-announce.md new file mode 100644 index 00000000..a8807ccc --- /dev/null +++ b/examples/openspending/content/blog/2013-06-30-spending-data-party-announce.md @@ -0,0 +1,48 @@ +--- +authors: +- anders +redirect_from: /2013/06/spending-data-party-announce/ +title: Join us for a City Spending Data Party and map the money near you! +--- + +More than half of the citizens of the world [live in a city](http://www.unfpa.org/pds/urbanization.htm). From schools to fire brigades cities are responsible for some of the most important services we can think of when it comes to government. So whether the nearby school is paid by your city, council or [Município](http://en.wikipedia.org/wiki/Municipalities_of_Brazil) chances are that the service is operated near you and that everyone should have access to information about the money is being spent.
    + +We are therefore thrilled to announce the global **City Spending Data Party - July 19th to 21st**. + +**Where:** Add your city to [the list](https://docs.google.com/a/okfn.org/document/d/1Zh-TPxgMiFDrzk-rNJqL9CmCbbtlZmp2xjWlZ6T20TA/edit#heading=h.f0m4z18ez212) + +**Who:** You, other spending explorers and the rest of the OpenSpending community + +The Spending Data Party came together thanks to feedback and engagement from [Hasadna](http://www.hasadna.org.il/en/) (Israel), [OpenBudgetOakland](http://openbudgetoakland.org/), [OpenKratio](http://openkratio.org/), [BudgIT](http://yourbudgit.com/) (Nigeria), [OKF Japan](http://spending.jp/) and [OKF Nepal](https://twitter.com/okfn_np). With the City Spending Data Party we want to open budget and spending data from cities to help us all get a sense of where the money is being spent. We hope that you will join to open up the budget and spending of your city! + +Register for the [**City Spending Data Party** ](https://docs.google.com/a/okfn.org/forms/d/1uHNAh9cfP_F5nudGfhV8t0XnNPmFL7cXhkuqWHAMaR4/viewform)! + +###Explore spending in your city - dive in
    +- Do you know how much your city actually spends on education compared to police enforcement? + +- Do you want more people to get people involved in the budget of your city?
    +Maybe you can learn a few tricks from [OpenBudgetOakland](http://openbudgetoakland.org/mayor_13-15_proposed.html) who created a site where citizens have been commenting on the budget proposal throughout the entire decision making process. + +- Which companies works for your city, and do they actually do a good job?
    +Cities such as [London](http://openspending.org/gb-local-gla) and [Washington DC](http://openspending.org/dc-vendors-contractors) release transactional spending data that allows you to see what companies clean up the garbage and repair the roads. Let's make many more cities open up their spending data! + +- Share what you learn with the community and ask if you get stuck! + +### What to do at the spending party? +- **[Load and visualise](http://openspending.org/datasets/new)** the budget of your city or province and add it to the list of [90 cities already mapped](http://apps.openspending.org/maps/) across the world. + +- **Open up budgets** from other local institutions such as [universities](http://openspending.org/blog/2013/06/17/universities-on-the-spending-map.html), public utilities or housing corporations. + +- **Learn from online trainings** how to visualise budgets, how to analyse spending data or how you can set up [a local OpenSpending satellite site](https://github.com/openspending/satellite-template). We will run various trainings and are eager to hear if you can contribute. Sign up [to do a training here](https://docs.google.com/a/okfn.org/forms/d/17diF5_alj37kvcY_2Oqx90xKz1aHJtjgVsXmL3yrHlY/viewform). + +- **Make OpenSpending better** by contributing to the code. We've got plans for some great enhancements and want to hear what you'd like to make. + +Need inspiration? Head over to the [Idea pad](https://docs.google.com/a/okfn.org/document/d/1Zh-TPxgMiFDrzk-rNJqL9CmCbbtlZmp2xjWlZ6T20TA/edit#), and see what cities have already signed up. + +###FAQ: +-I don't know where to get budget or spending data from my city?
    +Fear not! We'll make sure to share guides on how to access budget and spending data from your city and will also organise a **Data Clinic on July 10th**, where we'll help you search for data or ask your city for it, if it's not available. +
    +Would you like to join us mapping the money near you on **July 19th to 21st**? Register for the **City Spending Data Party** below! + + diff --git a/examples/openspending/content/blog/2013-07-02-project-and-community-proposal.md b/examples/openspending/content/blog/2013-07-02-project-and-community-proposal.md new file mode 100644 index 00000000..656b7d01 --- /dev/null +++ b/examples/openspending/content/blog/2013-07-02-project-and-community-proposal.md @@ -0,0 +1,14 @@ +--- +authors: +- anders +redirect_from: /2013/07/project-and-community-proposal/ +title: Supporting growth of the OS project and community - a proposal +--- + +Over the last few years OpenSpending has seen rapid growth in terms of technology, datasets, and community. As with many projects when they achieve a certain threshold of success and activity, the time has come to bring a bit more structure to the growth of the community and development of the project in order to empower more explicitly and more formally the growing array of stakeholders in the project.
    + +OpenSpending has always a been a community project. This proposal seeks to reflect this more formally in the governance and organization. The main proposed action is to establish a steering group to oversee the project and represent the growing number of stakeholders. In addition, there is a plan to establish specific “teams” who are looking after particular areas, in particular, a “technical (code) team” and a “data team”.
    + +We emphasize that the project will continue to have a home, legally and infrastructurally, at the Open Knowledge Foundation and the Foundation will continue to be strongly committed to the project. + +This information along with further details is avalable in [this public and commentable google doc](https://docs.google.com/a/okfn.org/document/d/12FVxSQQBovNGnGdtujecFoRs54LBqgp4zDgH0tdeQdk/edit#heading=h.19te6xx0f1ol) - please add comments there or share your feedback on [the OpenSpending mailing list](http://lists.okfn.org/mailman/listinfo/openspending). diff --git a/examples/openspending/content/blog/2013-07-03-New-Features-in-June-2013.md b/examples/openspending/content/blog/2013-07-03-New-Features-in-June-2013.md new file mode 100644 index 00000000..5cea177a --- /dev/null +++ b/examples/openspending/content/blog/2013-07-03-New-Features-in-June-2013.md @@ -0,0 +1,70 @@ +--- +authors: +- tryggvib +redirect_from: /2013/07/New-Features-in-June-2013/ +title: New Features in June 2013 +--- + +![Image by GemiTux (cc-by 2.0)](http://farm1.staticflickr.com/76/196956563_150aee58c0_z.jpg "Outside of OpenSpending!") + +The summer is here so shouldn't we be outside? Well, in a way we have. Bulk of the work this month has been on stuff **outside** of the core OpenSpending platform although there have also been some changes to the platform itself. + +### Contribution Attribution + +First up, and really important to us: **List of Contributors!** + +We added a [CONTRIBUTORS file](https://raw.github.com/openspending/openspending/master/CONTRIBUTORS) to the root of the openspending repository, just like you would expect in an open source project like openspending. + +If you think your name should be there, we didn't intentionally leave it out. We let *git* list our contributors but unfortunately that's only code contributors, there are plenty of other contributors and we want them in our list as well. Let us know if you are missing from the list and we'll add you! + +### Better Documentation + +This might seem like a small change, but it's an important one. Our *[installation docs](http://docs.openspending.org/en/latest/install.html)* were slightly wrong, which might have caused users some problems. They also referred to an outdated version of *solr*. + +That's been fixed now so we've reduced some frustration involved in getting an instance of OpenSpending up and running. If you notice any places in our docs where improvements are needed let us know, or better yet help us by contributiong improvements. + +This month we also released some much needed documentation regarding our development process, to help new contributors (and older ones) to know their way around project contributions. We created a short *[Howto hack on OpenSpending](http://openspending.org/help/hacking.html)* as well as a more detailed documentation about the *[development process](http://openspending.org/help/development-process.html)*. For code reviewers we also documented some *[guidelines for our code review process](http://openspending.org/help/code-review.html)*. + +### Satellite Template + +Our [satellite template](http://github.com/openspending/satellite-template/) exists to make creation of satellite sites easier. It was created last month (and based on *Where Does My Money Go?*) but it has already been used to create (or initiate) new satellite sites. As always when a fresh piece of software gets used, unforeseen use cases turn up. + +We were notified of those problems and were therefore able to put effort into adapting the satellite template to these new use cases. Some of these changes have been added into the template but as more satellite sites pop up, more changes will most definitely get added. + +We encourage you to use the satellite site if you are in the process of creating a satellite site, or want to create a satellite site. Also, let us know if you think we can improve the template. + +### Preparations for Inflations + +We have been working hard on making it possible to do *fair* historical comparisons with OpenSpending by adjusting for inflation. This is quite a large undertaking but nothing we can't handle. This month we've been laying the foundations for these inflation adjustments. + +First we had to collect some data. We decided to start small and start by looking at Consumer Price Indices (CPI) only. These give a pretty good indication about the inflation and are frequently used in economics. The data was collected and stored in standardised form as a [data package](http://www.dataprotocols.org/) and made available on [http://data.okfn.org/](http://data.okfn.org/data/cpi/). + +We went through a lot of CPI data available online and chose the best open data resource we could find. If you know of better data we could use, please help us improve the data, because better data results in better inflation adjustments on OpenSpending. + +Then we created a small module to read in data packages and called it [datapackage](http://github.com/tryggvib/datapackage). We created a fairly generalised module so that it could reused in other areas of OpenSpending or in other projects. This module, which we implemented in Python and made available in the [Python Package Index](https://pypi.python.org/pypi/datapackage/), almost instantly sparked off an equivalent [module in Java](https://github.com/rossjones/datapackage-java). Then later we received an improvement to our python module. All in the scope of one month. By the looks of it we succeeded in creating a generalised, reusable module. Well done! + +Using our datapackage module we then proceeded to build a first version of an economical transformation toolkit, which we dubbed *[economics](http://github.com/tryggvib/economics/)* (yes, we are very creative when it comes to naming our python modules). At the moment it can do basic inflation computations using the CPI data and we made that available in the [Python Package Index](https://pypi.python.org/pypi/economics/) as well. You can add more economical methods and computations if you like! + +Now we're set to start implementing inflation adjustment in the core OpenSpending platform. A huge thanks to all the economists, developers, data wranglers, and advisors for their help. July will be an exciting month! + +### Blog migration + +Another big change coming in July is a more standard blogging platform for OpenSpending. We have been using *[Jekyll](http://jekyllrb.com/)* to generate our blogs statically and serving those static pages via the OpenSpending platform. We decided to move our blog back to [WordPress](http://wordpress.com/). This will make blog contributions even simpler since many people are more comfortable with WordPress than markdown. + +We're not quite there yet (look for this change in July) but we called upon our task force to help us migrate the content and get the blog up to speed. We launched a *IRC hack session* where we collaborated on a [script to migrate Jekyll content to WordPress](https://github.com/tryggvib/jekyll-to-wordpress/). The content has been migrated but there are some UI/UX tweaks we want to do before we launch our main blog as a WordPress blog. + +If you know your way around WordPress and want to help. Let us know and we'll fill you in on what needs to be done. + +### FarmSubsidy + +In case you hadn't noticed we also launched and relaunched again the [FarmSubsidy project](http://farmsubsidy.openspending.org/) as part of OpenSpending. We initially launched an improved version of the project (after it was adopted by OpenSpending) on our servers but quickly noticed that we needed a dedicated server for the project. + +So we took Farmsubsidy down for a couple of days while we sorted out the server issues and moved it to a dedicated server. After reloading the data onto the new server we were able to relaunch Farmsubsidy so that the user experience should be better than it was in the first few days and now you can really start investigating how the European Union subsidises farms. + +### Thanks + +As always there were loads of other changes and happenings in and around the OpenSpending project. We would love to get some help to achieve even more in the coming months and like before we want to give a shout-out to all those who helped us in June. + +Thanks to **Michael Bauer**, **Gunnlaugur Thor Briem**, **Lucy Chambers**, **Velichka Dimitrova**, **Martin Keegan**, **Dan Lemon**, **Andy Lulham**, **Tom Morris**, **Prakash Neupane**, **OpenRotterdam**, **Florian Oswald**, **Daniel O'Huiginn**, **Anders Pedersen**, **Rufus Pollock**, **Niels Erik Kaaber Rasmussen**, **Joel Rebello**, **Todd D. Robbins**, **Nils Toedtmann**, **Stefan Wehrmeyer**, and **Guo Xu** for their contributions this month (there are probably a lot more who've contributed somehow to this month's features so sorry in advance if you're missing from the list). + +Image of window used for this blog posts is by [GemiTux](https://www.flickr.com/photos/gemitux/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). diff --git a/examples/openspending/content/blog/2013-07-04-exploring-senegal-public-procurements-how-we-turned-pdf-files-into-browsable-data.md b/examples/openspending/content/blog/2013-07-04-exploring-senegal-public-procurements-how-we-turned-pdf-files-into-browsable-data.md new file mode 100644 index 00000000..dfcea13b --- /dev/null +++ b/examples/openspending/content/blog/2013-07-04-exploring-senegal-public-procurements-how-we-turned-pdf-files-into-browsable-data.md @@ -0,0 +1,110 @@ +--- +redirect_from: /2013/07/exploring-senegal-public-procurements-how-we-turned-pdf-files-into-browsable-data/ +title: 'Exploring Senegal Public Procurements : how we turned PDF files into browsable + data ?' +authors: +- pierre chrzanowski +--- +Blog post co-authored by Patrick Nsukami (Dakar Linux User Group), Pierre Chrzanowski (Open Knowledge Foundation France) and Tangui Morlier (Regards Citoyens). + + + +As part of the “Tandem Dakar Paris” a Digital Week organized in May by the French Institute, Jokkolabs (Dakar) and La Cantine (Paris), people from Senegalese and French Open Data communities joint their forces during a workshop in Dakar to explore Senegal Public Data. + +During our exploration, we have been pleased to find that Public Procurement Data were regularly published on the Senegalese Authority for Public Procurement website. But the information were released in PDF files and therefore difficult to analyze. So we decided to extract the data into machine readable and open format. In a less than one day, 4800 lines of public procurements awarded between years 2012 and 2013 have been extracted into a single CSV file. It was then easy to upload the file onto Open Spending, making the data easily browsable. Here is how we did it : + +

    Downloading PDF files

    +The Senegalese Authority for Public Procurement releases public procurement data as PDF files on a quarterly basis + +armp + +The files contain information on each public procurement contract awarded during this period of time, including : + +
      +
    • amount of the procurement ;
      • +
    • date of the procurement has been awarded ;
      • +
    • public authority who delivered the procurement ;
      • +
    • private or public entity who received the procurement (the beneficiary);
      • +
    • type of procurement : service, good, etc.
      • +
    +We choose to download files from the first quarter of 2012 to the first quarter of 2013 (the last one available). + +

    PDF extraction

    +One of the tricky part was then to extract data from the PDF files. To transform a PDF file to a CSV file, it was important to split the task into small ones, and follow the KISS principle (Keep It Simple and Stupid). That's what we did, see it here : + +Task #1 - Split the PDF files : First step was to split the PDF into one-page-only files that would be easier to handle by scripts. This means to go from one document, containing lots of pages, for example 100 pages, to 100 documents of 1 page each. We used pdftk a free and opensource command lines tool, to perform this operation. + +Task #2 - PDF to XML : Second step was to turn the tables within the PDF files to an XML document. It is easier to parse an XML document (the structure) than a PDF one. For this task, we used pdftohtml + +Task #3 - XML to CSV : then we wrote a script in perl (but you can do it in python, ruby, etc.) that turned the XML document to a CSV document. Here is the XML to CSV  for 1 page script  + +The script creates the columns and the lines we defined. When we were satisfied with the result for the first page, we wrote another script to apply the first script to all the XML pages (a bash script). Here is the XML to CSV bash script + +Task #4 - Retrieve public procurement data only : The result was a CSV file containing all the information from the tables, including the data on Public Procurements but not only. The final step was to retrieve only data on "Public Procurement". This has been done by parsing the XML file and retrieving only lines with a specific content. The Retrieve script is here + + +Task #5 - Run all the scripts : finally, we wrote a bash script to perform all the tasks in a row : + +
      +
    • split pdf
      • +
    • turn pdf pages to xml ones
      • +
    • convert XML documents to CSV ones
      • +
    • retrieve all the lines on "Public Procurement"
      • +
    +Here is the final bash script + +Eventually, we got a pretty good first version of the csv file with all data on public procurements inside. Next step was to ensure the quality of the dataset and prepare it to be uploaded onto Open Spending. + +All the scripts created during this part are published in AGPL : http://www.gnu.org/licenses/agpl.html and can be forked from https://gitorious.org/sharedscrapers/aod_senegal + +

    Cleaning and preparing data

    +When extracting data from a PDF file, most of the time your csv comes with conversion errors such as misplaced or divided cells. It is important to correct those errors before going any further. Your dataset is a combination of different inputs from different public authorities : you must also ensure that every reference in the dataset has a unique name (public authority, company, etc.). This will facilitate the analysis. Finally, for uploading the dataset onto Open Spending, you must also ensure the quality of the data such as avoiding empty cells, or converting amounts and dates in the appropriate formats. All this is part of the cleaning process. + +To perform those operations we used only 3 kind of tools, linux shell tools, a spreadsheet application LibreOffice Calc (but you can also use Excel) and Open Refine (formerly known as Google Refine). + +Linux shell tools such as awk, grep or wc that help to identify the error and check if they are correctly corrected. They help to aggregate data and be sure that corrections needed were done globally. A text editor can also be used for this step. + +Open Refine is a powerful application to help you quickly clean your dataset. We mostly used Refine for its “cluster” feature, which helps you find groups of different cell values that might be alternative representations of the same thing, and then let you attribute a unique name to those cells. We used the “cluster” tool once we were sure that the cells contained the correct information. + +refine + +And we used LibreOffice Calc for all other operations : aggregate divided cells ; convert amounts in the right currency ; fill in empty cells with a default value. + +Here is a summary of the tasks performed during the cleaning process : + +
      +
    • Fill in empty cells with default values ;
      • +
    • Concatenate values that are divided into several cells ;
      • +
    • Attribute a unique value to same entities (public authorities, companies) ;
      • +
    • Convert all money amount into local currency “Francs CFA” ;
      • +
    • Convert date format from DD/MM/YYYY to YYYY-MM-DD (standard for Open Spending).
      • +
    +

    Uploading data onto Open Spending

    +Once the csv has been cleaned, we published the dataset on NosDonnées.fr the French Open Data Hub, under the Odbl licence. The file was then available for all in an open and machine readable format. + +nosdonneesfr + +Last part was to upload and configure the dataset to be available on Open Spending : + +Step #1 - Import and validate the dataset : assign a name, an identifier, a country, language, a currency and a description to your dataset. Then specify the url of your dataset. The file must be available on the web, this is why we published it first on NosDonnées.fr. + +import + +  + +Step #2 - Create a model for your dataset : once the file has been validated by Open Spending, you are invited to attribute dimensions related to your spreadsheet, i.e defining which field designate what. The mandatory fields are the field date, the field amount, the field “to” which is the entity who received the contract, and the field “from” which is the “entity” offering the contract. + +Fields “to” and “from” must be compounds dimensions, however the model will not be valid. Make also sure you have an unique ID for each entry. In our case we had to assign an unique code for each procurement row since it was not present in the original file. + +You can also specify which dimensions you want to be able to browse by in the Open Spending application. These are the facets of your model. We choose “public authorities”, “beneficiaries” and “type of procurements” and “procurements titles” as facets for our model. + +Once the model was ready, we saved it. Then, Open Spending started loading the entire dataset. In case of errors during the loading process, you can always come back to your dataset, do a bit of refine and reload it. After some trials and errors, the result was good enough to be published. + +Step #3 - Create a view : Dataset was now browsable. Last step (which is not mandatory) was to create specific views or visualisation. On Open Spending you can choose among “Table of aggregates”, “Bubble tree” and “Tree Map”. In our case we wanted to see a list of beneficiaries sorted by who received the most. So we choose “table of aggregates” with a view aggregated by beneficiaries. Once the view was created, we could embed it in our website like this : + + + +  + +So that’s all ! Senegal Public Procurements are now available on Open Spending and easily browsable. We hope this article will be helpful for your exploration into other country's public procurements or PDF files. + diff --git a/examples/openspending/content/blog/2013-07-08-join-the-openspending-data-wranglers-and-help-open-up-finances-around-the-world.md b/examples/openspending/content/blog/2013-07-08-join-the-openspending-data-wranglers-and-help-open-up-finances-around-the-world.md new file mode 100644 index 00000000..a35d3216 --- /dev/null +++ b/examples/openspending/content/blog/2013-07-08-join-the-openspending-data-wranglers-and-help-open-up-finances-around-the-world.md @@ -0,0 +1,60 @@ +--- +redirect_from: /2013/07/join-the-openspending-data-wranglers-and-help-open-up-finances-around-the-world/ +title: Join the OpenSpending data wranglers and help open up finances around the world! +authors: +- Anders Pedersen +--- +

    Do you like sharing your skills for working with spending data? Would you like to help other community members to clean budget data? We are looking for volunteer Data Wranglers to prepare and load data into OpenSpending and to support community members with data-related issues. + +

    +

    How it works

    +

    As a Data Wrangler you will help prepare and upload data to OpenSpending. You’ll work on things like: + +

      +
    • +

      Clean the data: Spending data often suffer from being messy and full of peculiar errors. You will be there to give the data some care and attendance with tools such as Open Refine to fix the mess. + +

      • +
    • +

      Sanity check: Sometimes a sanity check from you is needed to confirm that the numbers add up and that the budget actually makes sense. + +

      • +
    • +

      Format the data: You will need to fit the data for the OpenSpending load function. + +

      • +
    • +

      Keep track of spending data waiting in line at the Progress Page. At OpenSpending we would like to upload datasets quickly after their discovery. + +

      • +
    • +

      Share tips and tricks on how to work with spending data with the community of users at our Data Clinics, trainings and Spending Data Parties. + +

      • +
    +

    Qualifications

    +

    The most important requirement is enthusiasm! In addition, if you know how to use a spreadsheet and what a CSV file is that’s more than sufficient. We have a strong community who can help you grow your data skills. Programming skills and detailed knowledge of budgets is a plus but not required;-) + +

    We will provide a short induction on how to work with OpenSpending and you will also have your peers in the data team to help out. + +

    What does it take?

    +
      +
    • +

      For joining the Data Team as Data Wrangler we ask you to commit 3-5 hours per month. + +

      • +
    • +

      We generally coordinate the work in Spending Data sprints, where you and the team upload a series of datasets within a designated period of time (e.g. 2 weeks). By working in sprints you can contain the required workload, while still harvesting the benefits of working closely with fellow data wranglers via IRC, Skype and Hangout. We emphasize that work can also be done asynchronously -- so if, say, you only have an hour free every Sunday at a particular time, that’s absolutely fine too! + +

      • +
    • +

      We will feature your profile on the OpenSpending site along with the rest of the Data Team. + +

      • +
    +The first training session for our team of Data Wranglers will take place on Thursday 18 July, so register your interest as soon as possible! + +

    To apply to become an OpenSpending Data Wrangler fill in the application form below and we will get back to you as soon as possible. For questions about what the role, get in touch with us at: info [at] openspeding [dot] org. + + + diff --git a/examples/openspending/content/blog/2013-07-09-community-update-from-openspending.md b/examples/openspending/content/blog/2013-07-09-community-update-from-openspending.md new file mode 100644 index 00000000..0769ed75 --- /dev/null +++ b/examples/openspending/content/blog/2013-07-09-community-update-from-openspending.md @@ -0,0 +1,44 @@ +--- +redirect_from: /2013/07/community-update-from-openspending/ +title: Community update +authors: +- Anders Pedersen +--- +This is an update with news from members across the OpenSpending community. We list some of the many ways you can get involved, and we give a status on how registrations are coming along for the City Spending Data Party on July 19-21. Hint, we have some great cities participating including Lagos, Minsk and Kathmandu! + +

    On the blog

    +Patrick Nsukami (Dakar Linux User Group), Pierre Chrzanowski (Open Knowledge Foundation France) and Tangui Morlier (Regards Citoyens) wrote up up a detailed account of how they liberated Senegalese procurements from PDF and published the data on OpenSpending. +Félix Ontañón wrote up a post about how he opened up university budgets in Spain. +Earlier this month we called for a discussion about the continued growth of the OS project and community based on this proposal. We are still eager to hear your thoughts and comments about this. + +

    Fresh data on OpenSpending

    +Several fresh datasets have been added from Bosnie-Herzegovina, Japan, Brazil and Uruguay in this round of additions to OpenSpending. + +川口市平成25年度一般会計予算 July 9, 2013 + +Bosnia and Herzegovina July 8, 2013 + +Programa FDI Uruguay July 8, 2013 + +Inesc July 8, 2013 + +Execução do Orçamento Federal do Brasil - 2000-2013, 8 July 2013 + +

    Get Involved

    +We have got an abundance of activities and development going on. Let us know how you wish to be involved. + +You can help adding spending data by heading over to our Progress Page where several datasets are waiting to be cleaned and uploaded. + +We are looking for Data Wranglers who can help clean and add data to OpenSpending. Register now and you will be able to join the first training on July 18th. + +Do you code? Hal Seki from OKF Japan is heading up a community sprint to add a few improvement to OS before and during the Spending Data Party. Get in touch if you want to help out! + +

    Upcoming events and activities

    +Thursday July 11 19:00 CET / 18:00 GMT / 13:00 EDT: The weekly Data Clinic, where we offer community support on how to work with budget and spending data - via Google Hangout and IRC freenode channel #openspending. +Bring your own spending data - or have a look at our Progress page to find one of the spending data sets we're currently working on. Register for the Data Clinic here or drop in at IRC + +Thursday July 18 19:00 CET / 18:00 GMT / 13:00 EDT: The Community GHangout will include updates from across the community and prepare for the City Spending Data Party + +Friday July 19 to Sunday 21 City Spending Data Party: Communities across the OS community will get together online during this weekend to map spending in their city. So far community members from Minsk (Belarus), Lagos (Nigeria), San Francisco, Oakland, Kathmandu (Nepal), Kampala (Uganda), Kota Tangerang, (Indonesia) and cities across Japan have registered for the event. Find out all the details about the event and how to register in our announce post. + +

    diff --git a/examples/openspending/content/blog/2013-07-10-where-does-my-money-go-activities-have-become-infectious-in-japan.md b/examples/openspending/content/blog/2013-07-10-where-does-my-money-go-activities-have-become-infectious-in-japan.md new file mode 100644 index 00000000..883bf44f --- /dev/null +++ b/examples/openspending/content/blog/2013-07-10-where-does-my-money-go-activities-have-become-infectious-in-japan.md @@ -0,0 +1,16 @@ +--- +redirect_from: /2013/07/where-does-my-money-go-activities-have-become-infectious-in-japan/ +title: "“Where Does My Money Go?” activities have become infectious in Japan" +authors: +- Hiroichi Kawashima +--- +

    “Where Does My Money Go? (WDMMG)” activities have become infectious in Japan, as sites are gaining ground across cities all over Japan. Accordingly, Open Knowledge Foundation (OKF) Japan Group has been receiving increasing number of inquiries on how to develop WDMMG sites from both forward-looking local government officials and interested individuals. + +

    In a year, since the first release of the first WDMMG site for Yokohama in July 2012, the number of WDMMG participating cities has jumped to 19 cities by July 2013. Enthusiastic engineers are forming a community of practice led by Hal Seki, Georepublic Japan, to share know-hows and experiences on WDMMG development and helping new members to become able to develop new WDMMGs for their interested cities. National media outlets such as Nikkei, Asahi and NHK have covered the growing WDMMG activities as a new phenomena caused by Open Government Data movement in Japan. + +Building on this growing attention to WDMMG, the Open Knowledge Foundation (OKF) Japan Group is planning to conduct Spending Data Party on July 20th and 21st to further assist interested city officials, engineers and civil society activists to create their target city’s WDMMG sites. This event will be held as a part of Global Spending Party organized along with the Openspending community. + +The event in Japan will highlight some advanced WDMMG-like activities in Japan, help recently joining players to develop their own sites, and discuss the latest issues on WDMMG development such as budget data standardization and transactional data collection. Please take a look at the WDMMG site for Japan and keep your eyes on new developments! + +名称未設定 + diff --git a/examples/openspending/content/blog/2013-07-11-opening-up-municipal-spending-data.md b/examples/openspending/content/blog/2013-07-11-opening-up-municipal-spending-data.md new file mode 100644 index 00000000..a9dbce35 --- /dev/null +++ b/examples/openspending/content/blog/2013-07-11-opening-up-municipal-spending-data.md @@ -0,0 +1,55 @@ +--- +redirect_from: /2013/07/opening-up-municipal-spending-data/ +title: Opening up municipal spending data +authors: +- Niels Erik Kaaber Rasmussen +--- +

    The open data web-agency Buhl & Rasmussen has developed a site visualizing the budgets of all 98 Danish municipalities for one of the biggest Danish news sites, Politiken. + +

    Municipalities are central for the functioning of the welfare state Denmark. They take care of a range of important tasks like social- and health care, primary education, social benefits, traffic and much more. However even in a year with local elections they do not attract much public attention. + +

    One reason to this might be that the barriers for ordinary citizens to engage in local politics are too high. One way to lower the barriers might be to make it easier to understand the most central decision made by municipalities each year: their budgets. + +

    The data + +

    There are comprehensive data openly available on Danish municipal budgets and accounts. Budgets and accounts are structured in a hierarchy with 4 levels and roughly 250 possible expenditure and income posts. Data for all 98 municipalities can be obtained from Statistics Denmark and dates back to 1978. Obviously a lot of changes has taken place since 1978 but historical data for the last 5 years or so are reasonable comparable with today’s figures. + +

    Reducing complexity - preparing the data + +

    Multiple problems arise when comparing historical accounts to the latest budget, as I decided to do for this project. First inflation must be taken into account. Secondly the responsibilities of the municipalities are not fixed over time and thirdly accounting practices changes over time. + +

    I choose to adjust time series using a combined consumer price and wage index. While this does not fix the problem with changing responsibilities and accounting practices it improves the overall comparability. + +

    Dealing with national reimbursements + +

    Some expenses paid for by local government are reimbursed in part by national government. Shall the reimbursed part be included as an expense or not? I decided to do both(!). When presenting the budget using the bubble diagram all figures represents expenditure minus related income if any. In addition to the bubble diagram a table view shows the budgets with income and expenditures separated from each other. + +

    In the budgets expenditure posts are split in operational and construction expenses. The separation makes good sense for a lot of analyses however for simplicity I choose not to differ (the separate parts of an expenditure post are shown with mouseover). + +

    I made an effort to include not just spending data but also figures describing the different sources of income. Openspending makes such a good case in highlighting public spending but seems to ignore income. To get a reasonable understanding of municipal budgets you’d have to consider both spending and income. + +

    The solution + +

    The main visualization is the well-known bubble chart from OpenSpending slightly altered to include an information box, when users click at the lowest level. The popup includes information on historical spending, related income and presents the expenditure as part of the total budget, divided by the number of inhabitants and as compared to national average. + +

    Information popup showing historical data, expenditure as part of the total budget, expenditure per inhabitants and expenditure compared to national average. + +

    The bubble chart itself is an inspiring tool - thanks to the Open Knowledge Foundation and the OpenSpending-project for providing it. + +

    In addition to the bubble chart I created some features useful when dealing with multiple budgets with same structured format. For users interested in a specific expenditure post, that post can be selected and a list of the municipalities spending the most/least per inhabitant (of in percent of the total budget) can be shown. + +

    The standardized structure of the budgets also made it possible to build a function to compare two different municipalities in details. + +

    +Comparing two municipalities. The small charts shows historical data. + +

    The compare-feature makes it easy for one to see how the neighbour municipal prioritize differently. + +

    Municipal elections are going to be held in Denmark later this year. Hopefully the opening of the budgets will help lowering the barrier for citizens to engage in discussions. + +

    Local spending in your city + +

    While data on budgets and accounts for municipalities in Denmark are relatively easy to obtain this is far from the case in every municipal or city around the world. Some places the data is not published at all, many places the data is not published in a machine-readable format and seldomly is data made available in a manner that makes it possible to compare one city to another in a meaningful way. + +

    The OpenSpending project is organizing a City Spending Data Party on July 19-21 - if you’re interesting in local spending data this is a great chance to get involved. + diff --git a/examples/openspending/content/blog/2013-07-26-city-spending-party-around-the-world.md b/examples/openspending/content/blog/2013-07-26-city-spending-party-around-the-world.md new file mode 100644 index 00000000..c822f869 --- /dev/null +++ b/examples/openspending/content/blog/2013-07-26-city-spending-party-around-the-world.md @@ -0,0 +1,42 @@ +--- +redirect_from: /2013/07/city-spending-party-around-the-world/ +title: City Spending Party around the world +authors: +- Anders Pedersen +--- + + +nepal-spending-party + +Last weekend more than a hundred budget nerds and engaged citizens gathered at 20 spending data parties to open up city spending and budgets. From Lagos to Kathmandu groups dived into budget data across the OpenSpending community as part of this first global City Spending Data Party from July 19 to July 21. The spending parties helped bring the number of cities on OpenSpending to 119. Here is our wrap up from the City Spending Data Party across the OpenSpending community. + +###Kathmandu +In Kathmandu, Open Knowledge Foundation Nepal organised a two day spending party for civic coders, journalists and students focusing on Kathmandu Metropolitan. Coders opened up the city budget from PDF-documents, while tackling tough language issues as documents were not published in nepali unicode. The team detected significant year to year rises in Infrastructure and Development spending as well as in the salaries of employees. On the last day for than 20 participants and journalists attended the presentation of the work. Read the full report from the Spending Party in Kathmandu here. + +###Lagos +In Lagos the amazing visualisation team at BudgIT hosted a spending party to compare several years of expenditure and revenue data from the city. The team used the spending time to [code a tool](http://yourbudgit.com/openlagos/) that will help other groups to build more visualisations using data stored in OpenSpending. + +openlagos + +###Tel Aviv and San Fransisco +As part of the City Spending Party, [Hasadna](http://www.hasadna.org.il/en/) organised hackathons between Tel Aviv and the Jewish community of San Francisco. In Israel 120 participants worked in six teams on open data projects. One of the teams worked on the Open Muni budget, which is a new app for municipal budgets developed by Hasadna. In San Francisco 60 people worked on two projects, one of which was the Open Muni project around budgetary data imports and translations. +The Hasadna team also worked with David Zbikowski, Julian Martinez and Marc Joffe on loading spending data for California cities from the Public Sector Credit Solution's California City Credit Scoring web site. Julian even won the grand prize at Saturday evening's raffle: a trip to OKCon in Geneva! + +###Toronto +Toronto-spending-hackathon + +In Toronto Gabe Sawhney organised a hackathon focusing on the budget of the city as well as lobbyist registry data. The team parsed the 2012 city budget from a messy bunch of Excel files, and uploaded it to OpenSpending producing a treemap and as well as a visualization with d3.js (see dataviz above). Read the report from Toronto here. + +###Tokyo +The OpenSpending community of the Open Knowledge Foundation Local Group Japan is rapidly growing and threw a full scale one and a half day spending party at Yahoo Japan. There were 50 participants from across the country, including elected officials from three local prefectures around Tokyo. + +At the party 15 cities started loading data into OpenSpending and 8 cities completed their own Where Does My Money Go? site, which enables citizens to see where their taxes are spent locally. Koganei city created a spending site that invites citizens to comment on the budget, a model which was pioneered earlier this year by OpenBudgetOakland, another member of the OpenSpending community. The spending party also began taking on some of the more challenging issues. Japanese cities do not have coherent budget classifications the community will therefore begin to explore how budgets across cities can be compared. Coders at the spending party also made several contributions to the OpenSpending codebase and contributors from Open Knowledge Japan have taken on the important task of making the OpenSpending Satellite site more user friendly by adding several new features. The spending party was covered by the national broadcaster NHK and by the participants in their own Storify. + +###Is your city missing? + + + +Besides the events featured above, the City Spending Party also included numerous projects from members taking their city budgets into their own hands with the help from the OpenSpending community. From Minsk to Brazil community members worked on city budgets, and one week after the spending party dots are still being added to the world map of cities. You can find the full list of participating groups here. + +Is your city spending party missing from our wrap up? We still want to hear your spending party report! Or if you want to organise your own spending party, then [get in touch](mailto:info@openspending.org). + diff --git a/examples/openspending/content/blog/2013-07-29-predicting-city-bankruptcies-with-open-data-the-case-of-detroit.md b/examples/openspending/content/blog/2013-07-29-predicting-city-bankruptcies-with-open-data-the-case-of-detroit.md new file mode 100644 index 00000000..dab49b73 --- /dev/null +++ b/examples/openspending/content/blog/2013-07-29-predicting-city-bankruptcies-with-open-data-the-case-of-detroit.md @@ -0,0 +1,28 @@ +--- +redirect_from: /2013/07/predicting-city-bankruptcies-with-open-data-the-case-of-detroit/ +title: 'Predicting city bankruptcies with open data: The case of Detroit' +authors: +- Anders Pedersen +--- +This is a guest post by Marc Joffe of Public Sector Credit Solutions. + +Many have noticed that the United States last week was struck by its biggest municipal bankruptcy ever, when the City of Detroit declared bankruptcy. Less well known is the fact that Moody’s, the major credit rating agency, downgraded the City of Chicago by three notches at about the same time. + +Earlier this year, I used audited financial disclosures to estimate the risk of city bond defaults, which often accompany bankruptcies, in the state of California. The research was funded by a grant from the state, but its conclusions are mine and not those of any official agency. The goal was to see whether open data collected from so called Comprehensive Annual Financial Reports (CAFR), that US governments typically file as PDFs, and open analytics (or open economic modeling), can serve as an alternative to standard credit rating agency analysis. + +The model created during the research is openly available and designed to calculate default probabilities, where higher scores are worse than lower scores. Since 1940, the annual default rate for American cities has been 0.10%. In 2012 however, 2 out of 265 or 0.75% of California cities defaulted on their debt, and so that is the average score in the model. Scores substantially higher than 0.75% therefore represent heightened credit risk. + +A number of people have asked me how the scoring model would have treated Detroit (Michigan) and Chicago (Illinois), which are in other US states. Here is a my response. + +A Google spreadsheet containing the model is available here and embedded below. It is a modified version of our original model. I entered data from Detroit’s 2012 CAFR, which was published on December 28, 2012 and the Chicago’s 2012 CAFR which appeared more recently. Based on our open model Detroit’s probability score is 3.34%, which is worse than almost every California city in our survey. Chicago’s score is also pretty bad: at 1.77% it is worse than the score for Stockton, which was one of the two California cities to default in 2012. + +The main driver of Detroit’s high default probability score is its negative general fund balance. The ratio of Detroit’s general fund balance to general fund expenditure is -27%. As reported in our April working paper general fund exhaustion, which means very low or negative general fund balances, were associated with the Vallejo, Stockton and San Bernardino bankruptcies. The situation in Detroit provides further evidence that municipal bond investors and other stakeholders would benefit by monitoring this indicator. + +Although Chicago does not have a negative general fund balance, it has an annual general fund deficit and declining revenue, two of the four indicators that drive the default probability score. Chicago also has a relatively high ratio of interest and pension costs to total governmental fund revenues. When these uncontrollable costs become relatively high, bankruptcy is harder to avoid. + +Assessing government default probability rates based on open data is today a challenging task, as most cities publish this data in PDF-format. Getting cities to publish such data in machine readable format, would make such research a lot easier. For the OpenSpending community the the bankruptcy of Detroit also underlines the need for addressing not only spending, but also revenue flows and liabilities. + +Collecting, extracting and analyzing data from public financial disclosures can help us evaluate the credit risk of our local governments openly and transparent. This could be an important way of using the OpenSpending concept and platform. + + + diff --git a/examples/openspending/content/blog/2013-07-31-community-update-data-wrangling-list-and-moving-day-for-the-progress-page.md b/examples/openspending/content/blog/2013-07-31-community-update-data-wrangling-list-and-moving-day-for-the-progress-page.md new file mode 100644 index 00000000..6948473d --- /dev/null +++ b/examples/openspending/content/blog/2013-07-31-community-update-data-wrangling-list-and-moving-day-for-the-progress-page.md @@ -0,0 +1,65 @@ +--- +redirect_from: /2013/07/community-update-data-wrangling-list-and-moving-day-for-the-progress-page/ +title: 'Community update: Data wrangling list, moving day for our progress page and + more' +authors: +- Anders Pedersen +--- +It has been a few busy weeks around OpenSpending, and so it is time for another community update! + +###Introducing Datatoload: a place to list datasets that you would like to add to OpenSpending +Two months back we launched progress.openspending.org in order to make it easy to show and tell what datasets you are working on. Since the launch we however noticed that progress.openspending.org seems to be ill suited for actually tracking and collaborating on the uploading process, as the site does not offer a place to discuss the data but only list it. We therefore recently created the repo Datatoload on the OpenSpending Github site, which will replace progress.openspending.org going forward. + +With Datatoload you can now not only list your dataset, but also ask a questions to to other users and discuss how to work with your dataset. You can learn more about the easy steps for getting to use Datatoload in this guide. + +Please note: For you to use Github to track datasets going into OpenSpending does not mean that you will need any programming skills! + +###Welcome to new community members! +Are you sensing that the OpenSpending community is growing? Well, you are right! This week we have had a look at how you are using the site, and got quite thrilled from what we saw. + +- The last month: 79 public datasets were added by 78 users +- The last three months: 115 datasets were added by 138 users +- This is compared to 171 users throughout all of 2012. + +The City Spending Party was certainly a big part of this as we went from 91 mapped city budgets in early July to 135 cities today. +We are eager to take on the community growth and ask you to get in touch if you can help assist on blog editing, data wrangling or code review via info [at] openspending [dot] org. + +###New on the blog +- The bankruptcy of Detroit through the lens of open data: Marc Joffe from Public Sector Credit Solutions wrote this excellent piece on the fragile finances of Chicago and Detroit. +- How to visualise local budgets? Niels Erik Kaaber Rasmussen wrote about how he visualised Danish municipal finances using bubble charts. +- Getting stuck cleaning spending data? Tony Hirst from School of Data wrote an excellent guide for cleaning spending data with Open Refine +- Hiroichi Kawashima gave an exciting update from Japan and the many deployments of Where Does My Money Go? satellite sites there. +- The community members who took part in the City Spending Party provided stellar updates for our blog post touring city spending parties around the world. + +Are you doing work on spending data or did you just complete a budget visualisation that you would like to share? Write a guest post for the OpenSpending blog by getting in touch at info [at] openspending [dot] org + +###Sign up for a volunteer task in our brand new community Trello board +OpenSpending is first and foremost a community project, and we are thrilled about the growing engagement. We however need your assistance to move forward on the many new tasks for making OpenSpending a good place to come for beginners as well as budget nerds. We have created this public Trello board, which will help us manage our many task as community from translations to guest blog posting that are ahead of us. + +If you have not used Trello before, fear not, but simply read this ticket, which explains how you can get started contributing to the various tasks at OpenSpending, small or big, while using Trello. + +###A Datawrangling mailing list +Last week we welcomed our Data Wranglers for a first introductory call and our announcement of the Data Wrangling team is right around the corner. A key task for the Data Wrangling Team will be to help community members to clean, load and visualize data with OpenSpending. We would therefore like to offer a place where community members can ask all these questions about how to scrape, clean and load spending data. For this reason we have created the Data Wrangling mailing list, where you can join such a conversation and get tips and tricks for working on budget and spending data from community members including our fantastic the Data Wrangling team. + +You can join the list right here! + +###Events and hangouts +We have got an Introduction to OpenSpending, a Community Hangout and the fantastic OKCon coming up, so mark your calendar! + +####Introduction to OpenSpending Tuesday 6 August +Are you looking to get started using OpenSpending: either to add your first dataset or create a visualisation? We will be running an online training on 6 August, at 16:30 CET / 15:30 BST / 10:30 EDT - your time zone via Google Hangout + +Sign up for the introduction to OpenSpending in the Google form below: + + +####Next community Hangout on Wednesday 7 August +From August we will reintroduce the bi-weekly meeting schedule, and scale back from the weekly calls we ran while preparing for the spending party. This means that we will host Community Hangouts every two weeks dedicated to both community updates and a data clinic, where you can get help with the dataset that you are working on. + +We will organise our next Community Hangout on 7 August, at 19:00 CET / 18:00 BST / 13:00 EDT - your time zone via Google Hangout + +Sign up in the form below: + + +####OKCon in Geneva 16-18 September +OKCon will feature several budget and spending related sessions on topics such as on open finances and spending standards. Have a look at the entire schedule and featured speakers and get your ticket. We would be thrilled to see you there! + diff --git a/examples/openspending/content/blog/2013-08-02-lagos-city-spending-party.md b/examples/openspending/content/blog/2013-08-02-lagos-city-spending-party.md new file mode 100644 index 00000000..c16e9340 --- /dev/null +++ b/examples/openspending/content/blog/2013-08-02-lagos-city-spending-party.md @@ -0,0 +1,36 @@ +--- +redirect_from: /2013/08/lagos-city-spending-party/ +title: Lagos City Spending Party +authors: +- Oluseun Onigbinde +--- +

    Lagos is the financial capital of Nigeria. With a population of over 17 million people over an area of 356,861 hectares, Lagos is responsible for over 70% of Nigeria’s non-oil revenue.  Taken as a country, Lagos GDP of $86.7bn exceeds that of Kenya, East Africa economy powerhouse. The administration of Lagos State since the advent of new democracy cycle of Nigeria in 1999 has been under the national opposition party, Action Congress of Nigeria. + +

    With rising revenues from taxes and less dependence on resource rent from the federal accounts, Lagos is one of the three states in Nigeria which puts its budget online. Its financials in big headers in terms of expenditure, revenue and debt over the last years has also been available online for citizens to access. + +

    At Global City Spending Party organised by Open Knowledge Foundation, BudgIT, a Nigerian startup using creative technology to make data interesting, worked on Lagos released datasets. We combined our timeline framework with the OpenSpending tool to deliver what is very appealing. Using the Openspending tool was very important for the BudgIT team as it validates the recent capacity building project by OKFN which was funded by Indigo Trust. + +

    OUR VISUALIZATION + +

    We decided to build to show a yearly trend of Lagos revenue and expenditure datasets. The steps on the using the simple visualization are given below: + +

      +
    1. +

      Using the timeline within the PHP framework, users have the ability to see the Lagos revenues and expenditure over a period of years. + +

      2. +
    +
      +
    1. +

      On click of the timeline, users can see the relative sizes of the revenue and expenditure. + +

      2. +
    +  + +

    3. With an above instruction to click on the circle, users can peer at the data visualized with OpenSpending tool. + +

    Using a PHP framework with code available at https://github.com/olamieUsman/openlagos, the BudgIT team used a timeline framework to visualise the years and wrap the OpenSpending platform around the relatively sized bubbles. + +

    The application built exclusively by the BudgIT team over a weekend is an ongoing approach to make data more friendly, accessible and transparent for citizens across the literacy span. BudgIT maps  the array of tools such as desktop web, mobile web, sms, apps, infographics, interactive applications, print, radio, television, games and community groups for every specific access class. We continuously seek support of global organisations in making the Nigerian budget simple and understandable for every Nigerian citizen. + diff --git a/examples/openspending/content/blog/2013-08-05-kathmandu-metropolitan-spending-party.md b/examples/openspending/content/blog/2013-08-05-kathmandu-metropolitan-spending-party.md new file mode 100644 index 00000000..fc794dbe --- /dev/null +++ b/examples/openspending/content/blog/2013-08-05-kathmandu-metropolitan-spending-party.md @@ -0,0 +1,92 @@ +--- +redirect_from: /2013/08/kathmandu-metropolitan-spending-party/ +title: Kathmandu Metropolitan Spending Party +authors: +- Prakash Neupane +--- + + +Kathmandu, city of temple, capital of naturally enriched (Mountains, lakes, Hills, River) palace Nepal. Where people believe more than 4 millions (but record shows 2.5 millions) people reside in 50.67 square km area. All the governance body(ministries, departments, commissions), legislative body(Supreme Court, Armed Court), Security Force (Nepal Army, Nepal Policy, Armed Police Force) are centralized in Kathmandu. People have to come Kathmandu for going outside from Nepal, because only one international airport (Tribhuwan International Airport) is in Kathmandu. So it is obvious, the huge budget is spent for Kathmandu. + +Open Knowledge Foundation Nepal, organized City Spending Data Party on July 19-21. The objective was to work on the budget of kathmandu metropolitan that was announced just 1 week back. The program was invitation only , so the announcement for participation was done in 12 july http://np.okfn.org/2013/07/12/city-spending-data-party-kathmandu-2013/ . And finally selected 12 participants on the base of first come(this was for first two days while last day was public). + +As it was the first time event, every one worked hard, here is the summary: + +First Day: + +Everybody arrived on time, we discussed on what we are going to build, what could be the expected plan of three days. First task was group division, we have 4 groups + +Data Wrangler: Kshitiz, Manish, Sanjeev Khadka, Nikesh Balami, Rajan Silwal + +Code Hackers : Everest, Bidhya, Utshav, Rajendra + +Designer: Rajesh and Ashish + +Documentation and Social Media : Sajjan and Spandan + +Data Wrangler team download the pdf file of 22nd Kathmandu Metropolitan City council budget for fiscal year 2070/71from the official website of Kathmandu Metropolitan. + +Download some tools like tabular pdf to csv and tableau + +Differentiate the data into two topics, Aaya(Income) and Byaya(Expenditure) of Kathmandu Metropolitan + +Decided to work on Byaya (Expenditure), which include: + +Pujigat Kharcha(Capital Expenditure) + +Talab, Parisramik, Subidha Kharcha(Salary and Facilities Expenditure) + +Karyalaya Sanchalan Kharcha (Office Running Expenditure) + +Purwadhar Sudhar tatha anya bikash Kharcha (Infrastructure and Development Expenditure) + +  + +Second Day + +Day 2 kicked off with data wranglers completing remaining works of day first (completed total conversion) + +Code hackers and designers worked on the visual par + +Visualizing the information from the budget of Kathmandu Metropolitan city in simple charts and spending visualizations. + +Designs, banners and slides for next day + +  + +Third Day + +Around 25 participants including Former Minister, Media, Students and others + +We presented what we had made in last two days. + +DashboardDiscussed issues and problems we had faced while working. + +okfn + +Output + +We have written this post explaining the output. The output from openspending.org is at the top of the post or you can find it on OpenSpending. + +Impact and Learning + +The event was good learning for all, The first challenge was to extract the budget information from pdf, so it took almost whole day to properly extract the data from it. + +Second major problem was the language, the font was preeti, which means it was not in nepali unicode, the interesting thing is, Nepal's government has already address this through policy but due to the lack of proper management, it is not implemented in many place. + +Third was, lacking of technical knowledge of platform, later we studied and made it finally. + +After doing this event, Kathmandu Metropolitan appreciate our working and suggest us to give them feedback and also invite us there to resolve those issues. + +After this event, different social organizations, media are willing to collaborate us for the event like this. + +  + +Future Plan + +We are planning to continue this event in schedule(might be in every 3/3 months) + +We are planning to start a class to learn more technical skill with schoolofdata.org as well as a regular class with experts. + +We are going to work on whole government budget soon. + diff --git a/examples/openspending/content/blog/2013-08-12-openspending-news-round-up-august-12.md b/examples/openspending/content/blog/2013-08-12-openspending-news-round-up-august-12.md new file mode 100644 index 00000000..80763797 --- /dev/null +++ b/examples/openspending/content/blog/2013-08-12-openspending-news-round-up-august-12.md @@ -0,0 +1,42 @@ +--- +redirect_from: /2013/08/openspending-news-round-up-august-12/ +title: OpenSpending News Round-up, August 12 +authors: +- Teodora Beleaga +--- +

    Fiscal transparency never sleeps, and neither does the OpenSpending community. To keep track of all happenings across the open spending spectrum, we’re rounding up on latest blogs, stories and datasets each week. But we’re only human, so if we miss anything, give us a nudge at info [at] openspending [dot] org. + +

    Updates from the community + +OS Capture + +

    Last week saw several accounts of the City Spending Data Party we hosted in July, 19-21. Prakash Neupane shared the systematic way in which our community in Kathmandu, Nepal approached the city’s expenditure data, in addition to the more detailed account available on the Nepalese OKFN website. + +In Lagos, Nigeria the people at BugtIT visualised the city’s revenue and expenditure. Showing year-on-year changes from 2008 to 2013 this is a great example of functional art. Oluseun Onigbinde blogged about how the team in Lagos put this together. + +Further accounts of the City Spending Party are still to come. If you were part of the project and are yet to share your experience (Toronto and Tokyo, Tel Aviv or San Francisco, we mean your amazing work) please get in touch at info [at] openspending [dot] org. + +In the community, Michael Bauer from the School of Data experimented with Sankey diagrams made using d3.js and BOOST data from Kenya. Data journalist David Cabo also showed how he visualised budgets from Euskadi and Aragon, both Spain. + +Financial transparency around the world + +

    The Open Economics Principles were made available as part of the Open Economics project at the Open Knowledge Foundation. In Velichka Dimitrova’s words, Project Coordinator of Open Economics, they are “the guiding principles of transparency and accountability in economics that would enable replication and scholarly debate as well as access to knowledge as a public good.” Add your endorsement to the principles here. + +Slovenia’s public web portal designed to monitor government spending and expose corruption has won the prestigious UN Public Service Award. + +Marc Joffe, principal consultant at Public Sector Credit Solutions and Ian Makgill, managing director of the Spend Network, both part of the OpenSpending community, reviewed at the Guardian how local municipality projects work to open up spending, referencing other great projects such as OpenBudget Oakland and Open Muni from Hasadna. + +Interested in the Oakland Community Democracy project instead? The OpenSpending community found this article worth a read. + +The Sunlight Foundation and Code for America have teamed up to produce what they hope will be the most comprehensive survey of local procurement processes. Working in local US government? Give them a hand by filling in this form. + +New datasets on OpenSpending Datasets + +EU Financial Crisis Aid as donated by country from 2008 to 2011 has been uploaded and visualised on OpenSpending. + +datatoload is the place where we track government spending data that yet has to be loaded into OpenSpending. We have data from Massachusetts and New York to Finland and Moldova awaiting to be cleaned. Kudos to our team of data wranglers for claiming datasets and working their magic at unlocking data from pdfs and cleaning spreadsheets up. Want to join them? Get in touch at the data wrangling mailinglist + +Did we miss anything? + +While we strive to produce a comprehensive snippet of the fiscal transparency landscape across the world, this mammoth task gets the better of us every once in awhile. When that happens we trust we can rely on the wonderful community at the heart of OpenSpending to give us a nudge at info [at] openspending [dot] org. + diff --git a/examples/openspending/content/blog/2013-08-16-join-the-news-editor-team-at-openspending.md b/examples/openspending/content/blog/2013-08-16-join-the-news-editor-team-at-openspending.md new file mode 100644 index 00000000..78d8f29e --- /dev/null +++ b/examples/openspending/content/blog/2013-08-16-join-the-news-editor-team-at-openspending.md @@ -0,0 +1,39 @@ +--- +redirect_from: /2013/08/join-the-news-editor-team-at-openspending/ +title: Join our News and Website team! +authors: +- Anders Pedersen +--- +Every day, government spending makes headlines. At OpenSpending, we want to help the public put the numbers in perspective by showing how the news relates to spending data. **We are looking for volunteers to join our News and Website team to help run our blog and website and connect spending news with data.** + +As a member of the News and Web team, you will be involved in things like: + +* **Track down "spending stories"** – help find and report on the connections between stories in the media and the world of public finances - government budgets, spending, contracts etc – such as Detroit’s bankruptcy. Specifically: +* Collect and share news stories about spending and budgets. +* Find the spending data behind the news stories and write about it on our blog. +* **Edit the Blog**: oversee our blog with latest OpenSpending news including soliciting and editing contributions, writing roundups and news stories and more! +* **Oversee and manage our social media** – Facebook, Twitter etc. Explore new social media outlets (e.g. Tumblr). +* **Resources for journalists**. Write materials to help journalists and citizens dig into spending, understand tax (evasion), or create the next story about the EU budget + + + +### Qualifications + +The most important requirement is enthusiasm! Any experience with, for example, writing stories, editing a blog, running social media is great – but not required! + +We have a strong community who will support you and help you grow your skills. + +We will provide a short induction on how to work with OpenSpending and you will also have your peers in the News and Website team to help you out. + +### What Does it Take? + +* Time commitment: ~20-30 minutes a week on average = ~2h a month (note time can be irregular - its fine if you do nothing for a few weeks). Also if you have more time that's great - there's plenty to do! +* When to start: you can start immediately. +* You can also learn more about which tasks you will be doing at the Trello board for the News Team + +### Apply + +Apply by filling in the form below. We will respond to applications by August 23rd. + + + diff --git a/examples/openspending/content/blog/2013-08-19-openspending-news-round-up-august-19.md b/examples/openspending/content/blog/2013-08-19-openspending-news-round-up-august-19.md new file mode 100644 index 00000000..d51935c2 --- /dev/null +++ b/examples/openspending/content/blog/2013-08-19-openspending-news-round-up-august-19.md @@ -0,0 +1,40 @@ +--- +redirect_from: /2013/08/openspending-news-round-up-august-19/ +title: OpenSpending News Round-up, August 19 +authors: +- Teodora Beleaga +--- +

    Fiscal transparency never sleeps, and neither does the OpenSpending community. To keep track of all happenings across the open spending spectrum, we’re rounding up on latest blogs, stories and datasets each week. But we’re only human, so if we miss anything, give us a nudge at info [at] openspending [dot] org. + +

    Updates from around the community + +

    OpenSpending Community SnipWe’re expanding! The Open Spending community is on the hunt for News Editors to help spread the word about the outstanding work our members are engaged in, whether is cracking spending budgets in Nepal or putting together R-packages for our API. We are after volunteers who can turn fiscal and analytical geek speak into simple English and also have the ability to spearhead that into the social sphere. Sounds appealing? Then fill in this form by Friday, 23 August 2013. We look forward to having you on board! + +

    Another thing that’s got us all going right now - and do pardon us if you are subscribed to our mailing list and have already been receiving numerous references to this activity tracking tool already -  [but] the various trello boards we use to keep track of OpenSpending activities are simply worth another mention. Members across the community have already started to engage with the boards by joining activity cards or creating cards themselves. Kudos go particularly to the News Team and  the Translation Group for their volatility and responsiveness. Want to join us on trello too? Read this first. + +

    We are still awaiting several City Spending Party accounts and will share these with you as soon as possible. If you were part of the project and are yet to share your experience - Toronto and Tokyo, Tel Aviv and San Francisco, we are once again highlighting you and asking that you tell us about the great work in your city - [so] please get in touch info [at] openspending [dot] org. + +

    Financial transparency around the world + +

    As the decreasing US budget deficit hit headlines last week for reaching its lowest point in five years, the States’ national debt also got coverage for reaching no less that $16.7trillion. Sounds accurate? Did you visualise the US government spending? Are you planning to? Let us know in the comments below, at info [at] openspending [dot] org or by joining our mailing list. + +

    From spill to partnership, following Mexico’s President Enrique Peña Nieto proposals to loosen up ‘restrictive’ energy laws, the Wall Street Journal published a comprehensive Q&A post on what this means for both companies and consumers. + +

    Turns out ‘fiscal discipline’ is finally leading the eurozone out of crisis, or so Berlin and Brussels are accounted to saying in this FT article. Do you think the eurozone is out of recession? Let us know in the comments below, at info [at] openspending [dot] org or by joining our mailing list. + +

    Fear not! The Rupee is to be stabilised with a commitment to fiscal consolidation and a reduction in CAD (Current Account Deficit), was the Indian Finance Minister last week reported to have said. How does this news align with your analysis of the Indian budgets? Tell us in the comments below or join our mailing list and start the debate. + +

    CISCO, the systems and network company, released revenue data on their final quarter and fiscal year 2013. Not your usual budget data, but quite interesting nonetheless. + +

    A NIB that might be worth following up in order to track the exact source of spending cuts claims postmen and police community support officers are amongst those to be hit by a decreasing number of outdoor toilet facilities in Great Britain - allegedly caused by spending cuts.  What do you think? Ridiculous or outrageous? Either way, tell us in the comments below or at info [at] openspending [dot] org. + +

    New datasets on OpenSpending Datasets + +

    Last month saw the upload of tones of spending data from Moldova’s BOOST data to Catalonia’s latest budget and from Chennai’s City Budget to Portugal’s transactional spending data. Our team of data wranglers have been busy cleaning these up for you. You can track their progress and maybe give them a hand if you feel up to the challenge at https://github.com/openspending/datatoload/issues. + +

    Interested in #opendata? More from the foundation here. + +

    Did we miss anything? + +

    While we strive to produce a comprehensive snippet of the fiscal transparency landscape across the world, this mammoth task gets the better of us every once in awhile. When that happens we trust we can rely on the wonderful community at the heart of OpenSpending to give us a nudge. + diff --git a/examples/openspending/content/blog/2013-08-20-okfb-inesc.md b/examples/openspending/content/blog/2013-08-20-okfb-inesc.md new file mode 100644 index 00000000..848bb9f6 --- /dev/null +++ b/examples/openspending/content/blog/2013-08-20-okfb-inesc.md @@ -0,0 +1,79 @@ +--- +redirect_from: /2013/08/okfb-inesc/ +title: Visualizing How the Brazilian Government Underspends on the Public Good +authors: +- Vítor Baptista +--- +*This post was co-authored with Neil Ashton and is cross-posted from the [PBS Ideas Blog](http://www.pbs.org/idealab/2013/08/visualizing-how-the-brazilian-government-underspends-for-the-public-good).* + +Brazilian NGO INESC (Institute of Socio-Economic Studies) and Open Knowledge Foundation Brasil want Brazilians to participate in the allocation of their public spending and ensure that it is used to construct a free, fair, and sustainable society. + +That’s why we partnered to create [Orçamento ao seu Alcance][1], a site which presents the execution of the Brazilian federal budget in an interactive and intuitive form. + +We used OpenSpending as our database. This made it easier to focus and develop our visualizations without the need for setting up additional infrastructure for data hosting, and it made the data readily available in an accessible way. + +#### What's the project about? + +Millions of Brazilians pay the taxes that fund the federal budget, but few actually understand it. Most are unaware of Brazil's unjust regressive tax regime and of the scale of the losses to the public through misallocation. The information they need to understand these realities is simply not available in a comprehensible form. By building Orçamento ao seu Alcance, we hope to change that. + +Orçamento ao seu Alcance's development focused particularly on the issue of underspending. All Brazilian public bodies spend less money than is allocated to them, to varying degrees. The Ministry of Education, for example, [left 16.3% of its budget (about US$ 6.1 billion) unspent in 2012][2], and the Ministry of Culture only spent 47.5% of its budget in 2012. If Brazilians’ needs were really being met – if every Brazilian who wanted to study had access to good public schools, for example – this underspending would not be a problem. But that is far from the case; in fact [less than 1% of schools][5] [have an ideal infrastructure][6] (a problem we have [explored previously][7]). To explore and highlight the problem, we created a special-purpose data visualization. + +#### How we used OpenSpending + +Orçamento ao seu Alcance took data collected by [SIGA Brasil][8], an aggregator for the many systems used by the Brazilian government to organize budget data, and added it to the OpenSpending database. Using OpenSpending freed us from creating our own database and allowed us to use the OpenSpending API to construct visualizations and a full-text search system. + +##### Visualizing underspending + +We designed our own graph to tackle the problem of underspending. The result is a time series graph that combines bars, lines, and an area. The site constructs such a graph for each budgetary unit, showing how its budget and spending compare for a given year. + +Orçamento ao seu Alcance: underspending + +The blue area in the graph represents the total budget – which, as you can see, changes over the year. Each red *bar* shows how much was spent in a particular month, and the red *line* tracks total spending. The distance from the red line to the tip of the blue area gives the share of the budget remaining to be spent. The amount remaining in December is money that is *underspent*. + +This graph was built using [NVD3][9], a JavaScript library with a collection of reusable charts made on top of [D3.js][10]. The data comes from OpenSpending via its [Aggregate API][11]. + +##### Budget treemap + +For the index page, we wanted to show a broad view of the budget across all public bodies. More than that, we wanted to show the amount of money used in each function and subfunction (e.g. Education and Basic Education). To do this, we used the OpenSpending treemap visualization. + +Orçamento ao seu Alcance: treemap + +OpenSpending allows you to [create a treemap as a "widget"][12] which can be simply dropped into a site. We used a modified version of the widget code with customized colours and a "back" button for improved navigation. + +##### Searching + +To help the user find public bodies, we implemented a search box with auto-complete using[ Twitter Bootstrap][13]'s [typeahead][14] library. + +Orçamento ao seu Alcance: search + +To make the search instantaneous for the user, we load all data entries as soon as the user enters the page. The OpenSpending Aggregate API once again helped with this, allowing us to get a list of all public bodies with a simple query. + +#### Problems we had + +We did run into a few problems using OpenSpending to build the site, though all of them could be overcome. + +The Aggregate API only allows you to request one financial quantity (one measure) at a time. You can't request both a budget quantity and a payment at the same time, for example. Our underspending graph ended up using three measures, requiring three requests. This is a performance problem. Because the API caches results, however, it ends up being OK – and there are already plans to support multiple measures in future versions, so this problem will soon be solved. + +With the treemap visualization, our problem was that widgets are not customizable. They're made to be dragged and dropped into a blog post or a newspaper article, not integrated into a site with its own design. To change the treemap's colours and fonts, we had to use a modified version of the widget's code. + +#### Conclusions + +We're happy with how Orçamento ao seu Alcance turned out, and OpenSpending contributed a lot to its success. + +For developers, OpenSpending made it possible to run the site without its own database and to publish its content in a sleek, cacheable form. For the project's NGO supporters, using OpenSpending makes it possible to update the data without needing to deal with the site's developers. Everyone is happy. + +We hope that Orçamento ao seu Alcance will inspire other OpenSpending satellite sites that will help spread budgetary awareness around the globe. + +[1]: http://orcamento.inesc.org.br/ +[2]: http://orcamento.inesc.org.br/26000-ministerio-da-educacao/2012 +[5]: http://www.paraonline.com.br/menos-de-1-das-escolas-brasileiras-tem-infraestrutura-ideal/ +[6]: http://simaojacinto.blogspot.com.br/2013/06/menos-de-1-das-escolas-brasileiras-tem.html +[7]: http://www.escolaquequeremos.org/ +[8]: http://www12.senado.gov.br/orcamento/sigabrasil +[9]: http://nvd3.org/ +[10]: http://d3js.org +[11]: http://blog.openspending.org/help/api/aggregate +[12]: http://www.pbs.org/idealab/2013/03/how-to-embed-open-spending-visualizations-in-your-own-website078 +[13]: http://getbootstrap.com/2.3.2/ +[14]: http://twitter.github.io/typeahead.js/ + diff --git a/examples/openspending/content/blog/2013-08-20-site-move-sprint.md b/examples/openspending/content/blog/2013-08-20-site-move-sprint.md new file mode 100644 index 00000000..c93efeed --- /dev/null +++ b/examples/openspending/content/blog/2013-08-20-site-move-sprint.md @@ -0,0 +1,25 @@ +--- +redirect_from: /2013/08/site-move-sprint/ +title: OpenSpending site move sprint +authors: +- Neil Ashton +--- +We're doing a big move of OpenSpending web content! Most of is moving over to ****, and we need your help to get it there. + +We want to make it easier for the community to create and manage OpenSpending's web content. That's why we've created a new WordPress site at . We're going to make it the new home for all community-owned content – everything currently on openspending.org except for the OpenSpending system itself – and rename it **community.openspending.org**. The [help][1] and [about][2] pages have already been moved, with updates. + +#### Come help out! + +Give us a hand with the move! We're organizing a **site move sprint** for the week of **August 26 - 31**. Here's the plan and how you can get involved: + +* Check out the [site content audit spreadsheet][3], which explains what pages exist on the old site (Title, Current URL), what we think should be done with them (Action), where they should go on the new site (Dest URL), and whether anyone has dealt with them yet (Status). +* Please **jump in on the sheet**, mark some items you want to deal with (in the Owner column), and go to it. +* [Add yourself to the move sprint card][4] on the OpenSpending Trello to let us know you're participating. +* If you need access to the WordPress site, please [contact me][5] and I will hook you up. + +[1]: http://blog.openspending.org/help/ +[2]: http://blog.openspending.org/about +[3]: https://docs.google.com/spreadsheet/ccc?key=0AqR8dXc6Ji4JdEZpYWZoVWFWS2QxdkJCVzVoNmRodWc&usp=sharing +[4]: https://trello.com/c/GjLERmLu/54-openspending-site-move-sprint-august-26-to-31 +[5]: mailto:neil.ashton@okfn.org + diff --git a/examples/openspending/content/blog/2013-08-26-openspending-news-round-up-august-26.md b/examples/openspending/content/blog/2013-08-26-openspending-news-round-up-august-26.md new file mode 100644 index 00000000..48271b4e --- /dev/null +++ b/examples/openspending/content/blog/2013-08-26-openspending-news-round-up-august-26.md @@ -0,0 +1,35 @@ +--- +redirect_from: /2013/08/openspending-news-round-up-august-26/ +title: OpenSpending News Round-up, August 26 +authors: +- Teodora Beleaga +--- +

    Fiscal transparency never sleeps, and neither does the OpenSpending community. To keep track of all happenings across the open spending spectrum, we’re rounding up on latest blogs, stories and datasets each week. But we’re only human, so if we miss anything, give us a nudge at info [at] openspending [dot] org. + +

    Updates from around the community + +

    First things first, today marks the start of our big move of OpenSpending web content from http://openspending.org to http://blog.openspending.org and, as you may remember, you’re invited! Our chief technical writer and analyst, Neil Ashton is leading the move and has set-up this spreadsheet to track changes. Should you wish to collaborate on this project contact Neil directly so he can hook you up on the platform. + +

    Elsewhere in the community, we held a Google Hangout last Wednesday where members had a chance to play with a prototype app created in partnership with agency Journalism++ which aims to help citizens understand spending figures. We also discussed suggestions for the IMF on its consultation over the Fiscal Transparency Code (last updated in 2007). You can review our notes here and add your own by Friday, 30 August 2013. + +

    OpenSpending community member, and principal consultant at Public Sector Credit Solutions, Marc Joffe, received a well-deserved grant from the Sunlight Foundation for his work on opening up Californian fiscal data at a local government level. The project aims to continue to gather financial disclosures and process them into standardised data. Read more here and here. + +

    Lastly, as you would expect, the work to open spending data and visualise government budgets very much continued here last week. From Brazil, Vitor Baptista shared how they used the OpenSpending API to visualise Brazilian government spending. Take a look at their graphics here and also check out how they went about solving key challenges, including a limit to the aggregate API which only allows users to request a measure at a time -  all in his post here. + +

    Financial transparency around the world + +

    In the UK, the department for Business Innovation and Skills (in short BIS) shared with the OpenSpending community a tool they put together to help visualise spending - sounds familiar? - check out the beta version at: http://www.gist.cabinetoffice.gov.uk/. + +

    Elsewhere around the world, in Qatar, government spending was reported to be on the rise, up 2.2% to $48.9bn. In the US, small businesses turned out to be “poorly positioned” to acquire federal contracts, while in Tunisia, fiscal stamps fees were increased to tackle the budget deficit. + +

    We also came across this post on potential savings big data solutions can bring to government spending. While the study referenced focuses on the US, several ideas can be applied across other countries too. Any thoughts? Please share in the comments below. + +

    New datasets on OpenSpending Datasets + +

    There is still much data to be claimed from last month’s mammoth uploads of spending data from Moldova’s BOOST data to Portugal’s transactional spending data. Our team of data wranglers have been busy cleaning these up for you. You can track their progress here and maybe give them a hand if you feel up to the challenge. + +

    Did we miss anything? + +

    While we strive to produce a comprehensive snippet of the fiscal transparency landscape across the world, this mammoth task gets the better of us every once in awhile. When that happens we trust we can rely on the wonderful community at the heart of OpenSpending to give us a nudge at info [at] openspending [dot] org. + +

    diff --git a/examples/openspending/content/blog/2013-08-28-d3-sankey.md b/examples/openspending/content/blog/2013-08-28-d3-sankey.md new file mode 100644 index 00000000..5178708d --- /dev/null +++ b/examples/openspending/content/blog/2013-08-28-d3-sankey.md @@ -0,0 +1,194 @@ +--- +redirect_from: /2013/08/d3-sankey/ +title: D3.js Sankey diagrams with the OpenSpending API +authors: +- J. Félix Ontañón +--- +*This post is cross-posted from the [PBS Idea Lab Blog](http://www.pbs.org/idealab/2013/08/how-to-use-d3-on-the-openspending-api).* + +OpenSpending has a built-in set of visualizations – bubble charts, treemaps, and tables – which are useful for exploring how data is structured in levels. None of them, however, are really suitable for representing spending flows. + +Fortunately, users of the [D3.js](http://d3js.org) data visualization library have given us many examples of visualizations suitable for that purpose. The purpose of this tutorial is to show how easily D3.js can be used to visualize spending flows with OpenSpending data. + +#### Introducing D3.js and Sankey diagrams + +D3.js is a JavaScript library that creates *data-driven documents* (hence D3). Data visualizations are constructed with D3 by specifying a meaningful relationship between data and graphical elements. No manual fiddling with lines and boxes is required. + +D3.js has a huge and active community of users, and they have built a set of [example visualizations](https://github.com/mbostock/d3/wiki/Gallery). Some of these are incredibly useful for catching the eye with money flows: Sankey diagrams, chord diagrams (or circular networks), and map networks. + + ++++ + + + + + + + + + + + +
    OS & D3: Sankey diagramOS & D3: Chord diagramOS & D3: Network map
    Energy and consumption + +Sankey diagramUber Rides by Neighborhood + +Chord diagramFlows of refugees + +Map network
    +All of these examples are fully reusable: all you need to do to use them is to replace their underlying data with your own. + +In the following example, we will focus on Sankey diagrams, as they can represent more than two levels of flow. Sankey diagrams: + +

    +are typically used to visualize energy or material or cost transfers between processes. [...] They are helpful in locating dominant contributions to an overall flow. (Sankey diagram article on Wikipedia) + +
    +#### The Aggregate API + +To get spending data into D3.js, we can use the OpenSpending API, which gives us spending data in a form that can easily be translated into something D3.js understands. + +The key API for producing spending data visualizations is the *aggregate API*, which groups together entries in the dataset, sums up their values, and returns the result as a JSON object. + +An aggregate API call looks like this, where `` is the ID of an OpenSpending dataset: + + GET /api/2/aggregate?dataset= + +If no other parameters are included, all entries in the dataset are put in a single group, and the values of every entry are summed together. + +Things get more interesting when we add a `drilldown` parameter. This specifies a dimension of the data which will be used to split the set of entries. Each possible value of the specified dimension becomes a group of entries with its own subtotal. + +Let's drill down on the `programa` dimension of the `ugr-spending` dataset, for example, and look at the shape of the output: + + GET /api/2/aggregate?dataset=ugr-spending&drilldown=programa + + { + "drilldown": [ + { + "amount": 283175993.0, + "num_entries": 54, + "programa": { + "taxonomy": "programa", + "html_url": "http://openspending.org/ugr-spending/programa/422d", + "id": 1, + "name": "422d", + "label": "Enseu00f1anzas Universitarias" + } + }, + { + "amount": 64294001.0, + "num_entries": 52, + "programa": { + "taxonomy": "programa", + "html_url": "http://openspending.org/ugr-spending/programa/321b", + "id": 2, + "name": "321b", + "label": "Estructura y Gestiu00f3n Universitaria" + } + }, + { + "amount": 47967613.0, + "num_entries": 27, + "programa": { + "taxonomy": "programa", + "html_url": "http://openspending.org/ugr-spending/programa/541a", + "id": 3, + "name": "541a", + "label": "Investigaciu00f3n Cientu00edfica" + } + } + ], + "summary": { + "num_drilldowns": 3, + "pagesize": 10000, + "cached": true, + "amount": 395437607.0, + "pages": 1, + "currency": { + "amount": "EUR" + }, + "num_entries": 133, + "cache_key": "a3b56dc06b8a869ffa49b0ff063562798b073a3a", + "page": 1 + } + } + +The aggregate API returns an object with two fields, `drilldown` and `summary`. The latter contains information about the dataset, and the former is a list of different values of the drilled-down dimension and the sum of the spending values of all dataset entries with that value of the dimension. Each different value is an item in in `drilldown`, and its sum is its `"amount"`. + +We can also split the dataset by *combinations* of dimensions. This API call gives us a subtotal for each combination of `programa` and `to`: + + GET /api/2/aggregate?dataset=ugr-spending&drilldown=programa|to + +Using the aggregate API to construct D3.js visualizations means writing code to traverse the JSON objects returned by the API and to translate their contents into the form D3.js expects. + +#### Building a Sankey diagram + +Time for the full exercise! We will build a D3.js Sankey diagram from OpenSpending API, in the following way: + +* **Materials**: 2013 income and spending budgets for the University of Granada (UGR) at Spain. These datasets are titled `[ugr-income](http://openspending.org/ugr-income)` and `[ugr-spending](http://openspending.org/)` on OpenSpending. +* **Methods**: An [R script](https://github.com/openkratio/ugr-presupuestos/blob/gh-pages/src/sankey_conversion_openspending.R) that gets data from OpenSpending API and transforms it into a [D3.js Sankey diagram JSON input file format](https://github.com/openkratio/ugr-presupuestos/blob/gh-pages/src/ugr-sankey-openspending.json). +* **Results**: A [presentation page](http://openkratio.github.com/ugr-presupuestos/) embedding the Sankey diagram, OpenSpending treemaps, and raw data. + +The first step is to determine what we want to show in the Sankey diagram. Which relations should be displayed? How many levels of flow are appropriate for a suitable reading of the data? What's the story that you want to tell? + +Relying on the UGR income and spending budgets, we can imagine money flowing from the sources of income to the University and then the University spending this money. Attending to the budgetary structure, we finally choose a three-level Sankey diagram: + +* **Level 1**: Income budget broken down as “articulo” (economic classification) targeting to “Universidad de Granada”. +* **Level 2**: “Universidad de Granada” targeting the spending budget broken down into “programas de gasto” (functional classification). +* **Level 3**: “Programas de gasto” broken down into “capítulos de gasto” (economic classification). + +Notice that since the total amounts of the income and spending budgets are equal, both sides of the Sankey diagram have the same size. + +The second step is being able to get the data. As we explained above, OpenSpending has an API that allows us to retrieve data aggregated by measures and drilled down by dimensions. + +Getting the JSON data for the three levels of our Sankey diagram is as easy as follows: + + GET http://openspending.org/api/2/aggregate?dataset=ugr-income&drilldown=articulo + GET http://openspending.org/api/2/aggregate?dataset=ugr-spending&drilldown=programa + GET http://openspending.org/api/2/aggregate?dataset=ugr-spending&drilldown=programa|to + +This is a partial return for the second call. Notice that the data needed for the Sankey diagram are “labels”, “amounts”, and links between nodes. + + { + "drilldown": [ + { + "amount": 283175993.0, + "num_entries": 54, + "programa": { + "taxonomy": "programa", + "html_url": "http://openspending.org/ugr-spending/programa/422d", + "id": 1, + "name": "422d", + "label": "Enseñanzas Universitarias" + } + }, + /* Two more drilldown entries here. */ + ], + "summary": { + "num_drilldowns": 3, + "pagesize": 10000, + "cached": true, + "amount": 395437607.0, + "pages": 1, + "currency": { + "amount": "EUR" + }, + "num_entries": 133, + "cache_key": "a3b56dc06b8a869ffa49b0ff063562798b073a3a", + "page": 1 + } + } + +The third step is to produce the JSON input file format for the D3.js Sankey diagram. It has two components: links and nodes. Nodes are joined with links (i.e. arrows with variable width) and are represented as an array of labels, while the links component refers to an array with three members: source node index, target node index, and value (in this example, amount of money). The indexes in the links component refer to the position of each node at the node's component. Check the [final JSON input file for this UGR example](https://github.com/openkratio/ugr-presupuestos/blob/gh-pages/src/ugr-sankey-openspending.json) for further details. + +So the data for Level 1 has income “articulo” labels as source, a hardcoded “Universidad de Granada” label for target, and amounts as value. Level 2 starts with a “Universidad de Granada” hardcoded label as source, spending “programa” labels as target, and amounts as value. For Level 3, we have spending “programa” labels as source, spending “chapter” labels as target, and amounts as value. The provided R script automates the process of retrieving the data and transforming it into a Sankey diagram JSON input file. The code's comments clarify how it works. + +The fourth and final step is to create a web page to show the Sankey diagram. Fortunately, with a well formatted JSON input file, the official D3.js Sankey diagram example is fully reusable. We simply replace the JSON file with our own and enjoy the results. Some CSS and JavaScript variables can be tuned for controlling the colour palette or the width of the diagram—just check out the D3.js documentation. + +#### Conclusion + +We’ve shown how easy it is to take advantage of the aggregation methods of OpenSpending’s API to extend OpenSpending's default set of visualizations. D3.js is a powerful toolkit that gives us a better comprehension of budgetary data. An out-of-the-box D3.js visualization using OpenSpending as a data warehouse would provide a nifty boost to the OpenSpending project. In the meantime, take a look at [Michael Bauer’s openspending-sankey](http://mihi-tr.github.io/openspending-sankey/), which makes it rather easy to create D3.js Sankey diagrams for virtually every OpenSpending dataset. + diff --git a/examples/openspending/content/blog/2013-08-29-new-york-comptroller.md b/examples/openspending/content/blog/2013-08-29-new-york-comptroller.md new file mode 100644 index 00000000..4b49a10c --- /dev/null +++ b/examples/openspending/content/blog/2013-08-29-new-york-comptroller.md @@ -0,0 +1,17 @@ +--- +redirect_from: /2013/08/new-york-comptroller/ +title: New York State Comptroller Using Open Data and Models to Track Local Government + Finances +authors: +- Marc Joffe +--- +In the wake of Detroit’s bankruptcy filing, many are wondering which city will be next. Thanks to an innovative fiscal transparency project undertaken by the New York State Comptroller’s Office, it is unlikely that the next municipal bankruptcy will occur in “Empire” State. + +Earlier this year, the Office started publishing 23 fiscal stress indicators for local governments along with summary scores derived from a weighted average of these indicators. Nine of the indicators are fiscal metrics, such as the ratio of debt service expenses to revenue. The other fourteen indicators are “environmental factors” mostly taken from Census data. They include measures of property values, unemployment, and child poverty. The [scoring data](http://www.osc.state.ny.us/localgov/fiscalmonitoring/excel/AllDataWorkbook.xls) for 900 New York State local governments is available in a large Excel workbook. The Comptroller’s local government [fiscal distress scoring methodology](http://www.osc.state.ny.us/localgov/pubs/fiscalmonitoring/pdf/fiscalstressmonitoring2013.pdf) is also available. + +According to Nathaalie Carey, Assistant Comptroller for Local Government and School Accountability, the scoring system arose from a collaboration between statisticians and auditors working in the division. Their work began by investigating what had already been done elsewhere. One source of inspiration, ironically, was a fiscal scoring project in Michigan. Until fiscal 2009, the Michigan Treasury department published fiscal distress scores based on [ten indicators](http://www.michigan.gov/treasury/0,1607,7-121-1751_47023-171423--,00.html). We do not know why this effort was discontinued, but do note that the Michigan system showed Detroit as an elevated risk in the final 2009 survey. The New York team was also influenced by work done at the International City/Council Management Association (ICMA). After developing their fiscal stress model, department staff solicited comments from academics, think tanks, and municipal leaders. + +In addition to the stress monitoring system, the Comptroller’s office promotes transparency through its [Open Book New York](http://www.openbooknewyork.com/) web site. This site allows users to see revenue and expenditure breakdowns by category for each local government. It also provides state level spending and contracts by state agency. This raw data enables researchers and the general public to investigate government spending priorities and to see how much money is flowing to specific contractors. + +Overall, the New York State Comptroller’s Office is providing citizens with copious amounts of free, machine-readable government spending data. Further, they use a fully transparent methodology to boil down the statistics into a single, clearly understandable fiscal health score. We hope that other states, provinces, and central governments worldwide gain inspiration from New York’s example. + diff --git a/examples/openspending/content/blog/2013-09-02-openspending-news-round-up-september-2.md b/examples/openspending/content/blog/2013-09-02-openspending-news-round-up-september-2.md new file mode 100644 index 00000000..7d265f52 --- /dev/null +++ b/examples/openspending/content/blog/2013-09-02-openspending-news-round-up-september-2.md @@ -0,0 +1,38 @@ +--- +redirect_from: /2013/09/openspending-news-round-up-september-2/ +title: OpenSpending News Round-up, September 2 +authors: +- Teodora Beleaga +--- +

    Fiscal transparency never sleeps, and neither does the OpenSpending community. To keep track of all happenings across the open spending spectrum, we’re rounding up on latest blogs, stories and datasets each week. But we’re only human, so if we miss anything, give us a nudge at info [at] openspending [dot] org. + +Updates from around the community + +As we enter September we’re getting excited about OKCon, the annual OKFN conference taking place in Geneva on 16-18 September 2013. Highlights include a talk about the demand for open financial data (The World Bank), a panel on opening up procurement data (OKFN) and a workshop on what stories the public spending data should tell (Sunlight Foundation) - all taking place on Tuesday, 17 September 2013. Tickets information - should you wish to join - is available here. + +Community member, Félix Ontañón Carmona wrote a tutorial on producing spending flowcharts, in the form of sankey diagrams,  using d3.js and the OpenSpending aggregated API. This was a cross-post with PBS Ideas Lab Blog - a group blog by innovative new media thinkers including Stanford Knight Fellows and leaders from the MIT Center for Civic Media among many others. + +Elsewhere on the blog, Marc Joffe, OpenSpending community member, and principal consultant at Public Sector Credit Solutions, explained why New York is among the least likely cities to follow-up Detroit’s bankruptcy example and how the Empire State city has ensured avoidance of such crisis. + +Neil Ashton, our chief technical writer and analyst, has jump-started the move of OpenSpending blog content from http://openspending.org to http://blog.openspending.org, which should explain why you have been seeing seriously pixalated pictures and the like across the blog. Should you have the relevant technical skills and find the will to volunteer to help, please get in touch with Neil via twitter or by writing to our mailing list at openspending [at] lists [dot] okfn [dot] org. For spying purposes, we are tracing changes here. + +Financial transparency around the world + +Trading Economics published a rough visualisation of Indian Government Spending from 2004 to the first quarter of 2013 using data from the Central Statistical Organisation, India. This follows the news of the rupee’s “biggest fall in a decade” according to Quartz and the country’s slowest economic growth since 2009 according to Bloomberg. + +Public Spending UK released planned spending data for 2015 also accompanied by a selection of rough charts. You can download the data using their usual download tool at http://www.ukpublicspending.co.uk/download_raw. + +In the US, Slate followed up a NY Times article on elected officials’ (namely state comptrollers) investment dealings of taxpayers’ pension funds which they labelled “a match made in hell”. + +New datasets on OpenSpending Datasets + +Last week saw additional national budget data added from Japan, as well as city spending data uploaded from Lodz, Poland. + +There is still much data to be claimed from last month’s mammoth uploads of spending data from Moldova’s BOOST data to Portugal’s transactional spending data. Our team of data wranglers have been busy cleaning these up for you. You can track still their progress and maybe give them a hand if you feel up to the challenge at https://github.com/openspending/datatoload/issues. + +Did we miss anything? + +While we strive to produce a comprehensive snippet of the fiscal transparency landscape across the world, this mammoth task could easily get the better of us. Should that happen we trust the wonderful community at the heart of OpenSpending to give us a nudge at info [at] openspending [dot] org. + +  + diff --git a/examples/openspending/content/blog/2013-09-04-visualizing-data.md b/examples/openspending/content/blog/2013-09-04-visualizing-data.md new file mode 100644 index 00000000..1b6cac43 --- /dev/null +++ b/examples/openspending/content/blog/2013-09-04-visualizing-data.md @@ -0,0 +1,226 @@ +--- +redirect_from: /2013/09/visualizing-data/ +title: Visualizing Data in OpenSpending +authors: +- Michael Bauer +--- +*This post is cross-posted from the [PBS Idea Lab Blog](http://www.pbs.org/idealab/2013/09/how-to-visualize-data-with-openspending/).* + +OpenSpending is a database, analysis, and visualization tool for budget- and expenditure-related data. It acts as a data warehouse, giving you access access to budgetary data from around the world. + +One of OpenSpending's most commonly used features is its ability to easily visualize budgets in two different formats. In this tutorial, we will use data from the World Bank's BOOST portal and visualize it using OpenSpending. + +#### What you’ll need + +We'll use these tools: + +* [Open Refine](http://openrefine.org) - for converting the data + +* Mozilla Firefox or Google Chrome (Refine requires a browser, and it doesn’t work well with Internet Explorer) + +#### Obtaining the data from World Bank's BOOST portal + +The World Bank's BOOST portal aims at providing insight into the way countries spend their money. They have done this for World Bank grants in Kenya with the portal [kenya.wb-boost.org](http://kenya.wb-boost.org). In this tutorial, we’ll use this data and visualize it using OpenSpending. + +The first step in the process is downloading the data from the World Bank's BOOST platform. + +##### Walkthrough: downloading data from the BOOST platform + +1. Open [kenya.wb-boost.org](http://kenya.wb-boost.org) in your browser. + +2. It defaults to a map view, which is not optimal for our purposes. Switch to the pivot table view of the tool by clicking on the "pivot table" tab. + + ![image alt text](http://i.imgur.com/BENYBVR.png) + +4. The table view is initially empty because the system does not know how to aggregate information. + + On the left, there are the options for the data we want to have shown. Add more groups to the selection to allow for a finer picture. Do so by dragging them from the "Group Rows" box into the "Selected Groups" box. + + We want to have: + + 1. Top-level spending Unit + 2. Expenditure type + 3. Sector + 4. Economic Category + 5. Sub-Economic Category + + When you have these, click on the "Build Table" button in the middle of the grey area. This will take a while. + +9. Now download the data with the "Export to CSV" button on the center top. This will ask you to pick a folder in which to store the file. + +#### Preparing the data for OpenSpending + +Great - now we have the data, so let’s prepare it for OpenSpending! + +OpenSpending needs the data to be in a specific format to be able to use it. Specifically, OpenSpending needs one column for the amount spent, one column for the date of the transaction (in our case the year), and then more columns for other properties (such as the spending unit, the sector, the economic category, etc.). + +To be able to use the data we just downloaded in OpenSpending, we have to bring it into the necessary form. We will do this with a tool called Open Refine. Refine is made for cleaning data and converting data from one format to another. + +##### Walkthrough: converting data with Refine + +1. Start Refine by double-clicking on the Refine icon. Once it’s ready, a browser window will open and point to [127.0.0.1:3333](http://127.0.0.1:3333). + +3. First we need to create a project with our new data. Do so by clicking on the "Create Project" tab. + + ![image alt text](http://i.imgur.com/ywah1MP.png) + +4. Now choose the file we just downloaded and click "next". This will open the Preview tab. You’ll notice how the first row is empty and contains the years for the executed budget. + + ![image alt text](http://i.imgur.com/lhXsJJO.png) + +7. We want to have the years in the column headings. Let’s tell this to Refine. Enter "2" next to the box saying "Parse the next ____ lines as column headers". + + ![image alt text](http://i.imgur.com/5uD0hld.png) + +8. Now that this is fixed, let’s create the project. Click on the "Create Project" button on the top right. + +9. OpenSpending wants a single column for all the amounts and a single column for all the years. Right now we have a column for each year. No problem, let’s transpose it. + + Select the column options for the first column ("Executed 2003") and select "Transpose → Transpose cells across Columns into rows". + + ![image alt text](http://i.imgur.com/GwPi7Gh.png) + +11. Select "Two new Columns". Call the first Year and the second Amount. Also make sure to select the "fill down in other columns" option. + + ![image alt text](http://i.imgur.com/FUu89HP.png) + +12. Click on "Transpose" to do the transposition. + +13. Now we only have to remove the "Executed" in the year column. We do this using a "Transform" on the column. + + ![image alt text](http://i.imgur.com/vbKrT45.png) + +15. A menu pops up asking us for the "Expression" to transform. Expressions are like formulas in spreadsheets. + + The expression we want to use is `value.replace("Executed ","")` - this will remove the "Executed " from the cells. + + ![image alt text](http://i.imgur.com/ZJHjJbo.png) + +16. Next, we have to check whether all the years and all the amount fields actually have values in them. Let’s do this using a facet. Select "Facet → text facet" on the year column. + + ![image alt text](http://i.imgur.com/0yeuDim.png) + + This will open a Facet window on the left side. + + ![image alt text](http://i.imgur.com/lX5sKW9.png) + +19. There are 39 rows where the year is blank – let’s delete them. Select the blank rows by clicking on "(blank)". + +20. Now let’s remove the rows by selecting "edit rows → remove all matching rows" from the "ALL" column options. + +21. Remove the facet by closing the facet window with the little "x" on the top left. + +22. Now our dataset is ready for Open Spending! + + Let’s export it to CSV ("Comma-separated value"): + + ![image alt text](http://i.imgur.com/fpWEcvC.png) + +Congratulations! You have successfully prepared a dataset for visualization and analysis in OpenSpending. + +#### Uploading the data to OpenSpending + +The next step we need to take is to upload the Data into OpenSpending. Since OpenSpending does not support direct uploading of data, we’ll have to store the data somewhere on the web. This can be your own website - or in our case a community-driven data portal, [AfricaOpenData.org](http://africaopendata.org). + +##### Walkthrough: uploading data on AfricaOpenData + +*Note: this step is optional. You could also use a service like [Dropbox](http://dropbox.com/) or your own website to upload the datafile.* + +1. Go to [africaopendata.org](http://africaopendata.org) and create a profile. + +2. Log in. + +3. Now click on the "Add Dataset" button in your dashboard. + + ![image alt text](http://i.imgur.com/IBqUkC9.png) + +4. This will ask you some basic information for the dataset - a name, a description and so on. + + ![image alt text](http://i.imgur.com/uA5O4em.png) + +5. Click on Next to go ahead. + +6. Add a data file by selecting the "Upload a File" tab and choosing your file. + +7. Enter some information about the file and click on "Next". + +8. Now you can enter some further information. Then click "Finish". + +Congratulations, you have published a Dataset on AfricaOpenData! + +##### Walkthrough: loading data into OpenSpending + +Now that we’ve got our data online, we can work on getting it into OpenSpending. + +1. Go to [openspending.org](http://openspending.org). + +2. Click on the "Log in/register" tab and register a new account, or log in with a previously registered one. + +3. This will open your Dashboard. If you just signed up, this will be empty. Click on the big blue "Import a Dataset" button. This will lead you to a form for you to describe your new dataset. Fill it out. + + ![image alt text](http://i.imgur.com/PJ2VhhR.png) + +6. Click on "Next Step". This puts us into the dashboard for the project. To import a dataset, we have to do several steps. + + First, add a source. Click on the "Add a Source" button. Now go back to your uploaded dataset and find the download button (a sample can be found [here](http://africaopendata.org/dataset/boost-kenya)). Copy the link into the menu that popped up in Open Spending and click on Create. + +11. OpenSpending will now analyze the file and see whether it’s good to load. This will take a while - feel free to reload the page to see whether it’s done. When it’s done, click on the "create model" button. + + ![image alt text](http://i.imgur.com/J3DyHiG.png) + +13. Open Spending will have recognized most of the columns properly – except it wants a column called "time", and we have to create it. Delete the Year column by clicking on the "X" next to "year" in "Existing Dimensions". + +15. Now let’s add a dimension by clicking on the "New Dimension" button. This opens a menu. Select "date" and name it "time". + + ![image alt text](http://i.imgur.com/zmhcHDC.png) + +17. Click on "Add" to add this dimension. OpenSpending should have automatically identified the "Year" column as the column you want to have. + +19. So far so good. To display data more nicely, OpenSpending needs two dimensions labeled "from" and "to" that refer to who spent the money and for what. In our case, those would be *Top-level-spending-units* for "from" and *sub-economic-category* for "to". + + Let's do this similarly to how we added "time". Remove "top-level-spending-units" from the existing dimensions. Create a dimension called "from". + + ![image alt text](http://i.imgur.com/q6gQ21i.png) + +23. Now you can add all the attributes to "from" here. In our case, it’s just the one column. + +24. Now do the same for "sub-economic-category" and "to". + +25. To make sure it doesn’t double-load the data, OpenSpending needs to know how to determine whether a record is already there. This is done by specifying which combination of columns marks an entry as unique. + + To define this, select "Set Unique Dimensions". A popup will appear. Check all dimensions except "amount". + + ![image alt text](http://i.imgur.com/e0gm7M5.png) + +27. Great - now click on "Update" and then on "Save Dimensions". This should create a Model in OpenSpending - so it understands what your data looks like. + + Once we have created our model, we can load the data. Go back to the Dashboard and click the "Load" button next to your source. This will take a while and run through. Reload the page and check until the run is finished. Now we can open the dataset. Click on the link next to the "house" icon on the top. + + ![image alt text](http://i.imgur.com/r8y9ZYh.png) + +Congratulations! You have uploaded a dataset to OpenSpending! + +#### Creating a visualization on OpenSpending + +Now that we have our Dataset on OpenSpending, let's create a visualization. + +##### Walkthrough: creating a Treemap visualization in OpenSpending + +1. Click on the Visualization tab and select "create a visualization". + + ![image alt text](http://i.imgur.com/f0TRZjK.png) + +2. We want to create a TreeMap, so let’s select "TreeMap". This gives us the Visualization editor. It allows us to select the parameters by which the dataset is split and filtered. + + Let’s start with "Sector". Then add a new level and select "From"; add another one and select "Economic Category"; and finally add one more level and choose "To". "To" will be our most fine-grained information on where the money is spent. + + ![image alt text](http://i.imgur.com/LJMIttP.png) + +6. You can play around with the visualization in the bottom to get an idea what is going on. Once you’re happy, click on "Save or embed". + +8. Name your visualization and click "Save". + +Now you have a visualization, and you can go and embed it on your website. + +Stuck? Need further support? Visit [ask.schoolofdata.org](http://ask.schoolofdata.org)! + diff --git a/examples/openspending/content/blog/2013-09-10-workshop-at-okcon-human-centered-data-what-should-public-spending-data-tell-us.md b/examples/openspending/content/blog/2013-09-10-workshop-at-okcon-human-centered-data-what-should-public-spending-data-tell-us.md new file mode 100644 index 00000000..a244db2d --- /dev/null +++ b/examples/openspending/content/blog/2013-09-10-workshop-at-okcon-human-centered-data-what-should-public-spending-data-tell-us.md @@ -0,0 +1,27 @@ +--- +redirect_from: /2013/09/workshop-at-okcon-human-centered-data-what-should-public-spending-data-tell-us/ +title: 'Workshop at OKCon: Human Centered Data - What Should Public Spending Data + Tell Us?' +authors: +- Anders Pedersen +--- +IMGP4446 + +We are now less than a week away from OKCon, with an amazing programme from topics on open government to open finances, procurements and spending. If you are looking to go, but did not get your ticket now is the time! If you will be at OKCon and are into spending data, make sure not to miss this gem of a workshop: + +**Human Centered Data: What Should Public Spending Data Tell Us?** + +Kaitlin Devine and Julia Keseru from Sunlight Foundation will head up this workshop at OKCon on **Tuesday 17 September, 14:45 – 16:00** @ Room 14, Floor 2. + +The workshop will ask the basic question: If there were no legacy systems or arcane accounting mechanisms to determine the default structure of fiscal data released to the public, what questions should it be able to answer? + +Release of fiscal data (budgets, expenditures, audits) has increased in recent years, but is largely a function of how a government happens to format its budget or accounting systems. But is this indicative of how people external to the government think about government spending? Many citizens think about government spending along the lines of broad priorities or goals of government, such as those discussed in political debates (i.e. reducing homelessness, Iraq war spending). + +Very rarely do expenditures line up exactly with these goals and civil society organisations usually struggle to make financial information accessible and relevant to the public. This workshop will gather diverse users (and non-users) and distill the natural expectations of what this data should provide, and contrast it with what currently exists. + +Disclosure so far has focused on getting what the government collects to be public. Is it time to move forward with changing how they collect it in the first place? + +The workshop space can accommodate up to 20 people. To sign-up, express your interest in the topic and get in touch with Kaitlin and Julia, please write to publicspendingdata@okcon.org. + +Link to the workshop in the OKCon schedule. + diff --git a/examples/openspending/content/blog/2013-09-13-finance-data-standards.md b/examples/openspending/content/blog/2013-09-13-finance-data-standards.md new file mode 100644 index 00000000..a4d13f0a --- /dev/null +++ b/examples/openspending/content/blog/2013-09-13-finance-data-standards.md @@ -0,0 +1,40 @@ +--- +redirect_from: /2013/09/finance-data-standards/ +title: What public finance can learn from public transit about data standards +authors: +- Friedrich Lindenberg +--- +Tram in Gothenburg in 1962 + +Cross-posted from TabbFORUM, a discussion board on important issues affecting global capital markets. + +Last week, Marc Joffe published a piece describing the potential of standardized financial disclosure for municipal governments (“The Case for Muni XBRL: Bringing Municipal Financial Disclosure Into the 21st Century,” TabbFORUM). I fully agree with the core of his argument – that the availability of structured information about a city's accounts helps, amongst other things, to estimate its creditworthiness. However, Mr. Joffe suggested a standard called XBRL, which should, in my opinion, be avoided in order to bring some sunshine into municipal finances. + +XBRL has seen incredible adoption over the last few years, as more and more government agencies across the globe are following the example of the US FDIC and SEC and now require companies to release their accounts using the standard. A proposed DATA Act that is under consideration by the US Congress would mainstream XBRL as standard across the federal government. Mr. Joffe's argument is based on this success: As XBRL is already recognized by various regulatory bodies, it should be simpler to convince government to apply it to itself. This is a valid point, and clearly, data released as XBRL is greatly preferable to the traditional PDF reports issued by many agencies. + +Yet I believe that we can do even better – by putting greater emphasis on the requirements of data re-users rather than regulatory bodies and data publishers. While the XBRL consortium members are mainly data publishers and IT solutions providers, government transparency efforts should invest additional effort into considering the needs of other types of re-users, such as analysts, investors and financial journalists. + +Implementing XBRL is stunningly complex; accessing the right information inside an XBRL document requires specialist software – and the list of open source efforts that would make such technology widely available is not very impressive. + +Data formats matter because of the target audience: As part of the OpenSpending project, I've been involved in training journalists in the use of simple spreadsheet software – and, frankly, the prospect of introducing a non-technical audience to XML processing scares me. + +Rather than opening up data, we risk raising a new barrier, which will limit use of data that should be easy to reuse in the public domain. For a non-governmental organization, the cost of hiring developers to process XBRL would not be very different from the barriers imposed by having to extract information from PDF documents or by having to pay for the data directly. + +Essentially, we're closing off the data from those most interested in its meaning, in the interest of expressing all the complexity that our accounting systems are able to produce. + +The open data community has already seen some of the issues caused by supplier-driven standards in one of its greatest successes, the International Aid Transparency Initiative (IATI). IATI, an XML-based standard, is now used to publish information on 76% of all official development assistance globally. It is easy to imagine the benefits of having activity information from more than 180 development organizations released in a common format: Simplifying the analysis and comparison of projects will lead to much improved effectiveness in foreign aid. + +But the story isn't quite as simple. The complexity of the XML-based standard has sometimes failed to support the eccentricities of individual publisher's databases, leading to incomplete and divergent data. Virtually any IATI project that I'm aware of starts off by converting the data to a more accessible format, often either CSV files or web services based on JSON, a data format that can easily be processed by web browsers. + +A very different approach can be seen in another field, public transport. The demand for data in this domain was pushed forward by a singular killer application: Google Maps. In order to get their transit information included in Google's service, municipalities were asked to submit their data in a format called GTFS, the General Transit Format Specification. + +Unlike XBRL or IATI, GTFS relies on the Swiss Army knife of data: CSV. This proto-spreadsheet format has been around since the late ’60s, yet it still remains one of the most popular and widely understood means of data transfer. + +To express complex timetable information, GTFS defines a set of files – such as bus stop locations, operator agencies, routes and stop times. Some of these are required, others are optional. Any of them can be interpreted on their own, but of course they must be linked up to create a moving model of a city's public transit. Convergence on a common format is eventually achieved through the standard's thorough documentation, which assists implementers with explanations of each field’s semantics as well as many practical examples. + +This model of many loosely linked components is a simple but efficient method to masquerade the differences between public transit systems world-wide. Such flexibility will also be required to look at public finance across many national and regional accounting cultures; while corporate accounting may have converged globally toward US GAAP over recent years, half of the municipalities in my home country Germany still run on a system of single-entry bookkeeping that has seen little change since they invented the printing press. + +Still, a format like GTFS does not impede our ability to converge on the semantics of the data: taxonomies and charts of account can be expressed in CSV the same as in XML – and a wide variety of tools exist to check the conformance of released data to external standards. + +So why not get over the complexity rush and start simple: by releasing a couple of easily generated CSV files, and by having an incremental discussion about how these can be aligned to achieve the eventual objective of applying common models and comparisons. + diff --git a/examples/openspending/content/blog/2013-09-16-xbrl-vs-csv-the-csv-might-be-a-swiss-army-knife-but-it-is-the-wrong-tool.md b/examples/openspending/content/blog/2013-09-16-xbrl-vs-csv-the-csv-might-be-a-swiss-army-knife-but-it-is-the-wrong-tool.md new file mode 100644 index 00000000..2d133b97 --- /dev/null +++ b/examples/openspending/content/blog/2013-09-16-xbrl-vs-csv-the-csv-might-be-a-swiss-army-knife-but-it-is-the-wrong-tool.md @@ -0,0 +1,45 @@ +--- +redirect_from: /2013/09/xbrl-vs-csv-the-csv-might-be-a-swiss-army-knife-but-it-is-the-wrong-tool/ +title: 'XBRL vs. CSV: The CSV might be a Swiss army knife, but it is the wrong tool' +authors: +- Anders Pedersen +--- +Apply webbing to the frame - chair, 1936 + +In our ongoing debate about the pros and cons of XBRL and CSV, we are pleased that we are able to post this response from Charles Hoffman, who is widely credited as one of the main accountants behind XBRL. If this is the first post you read in the series do not miss the earlier entries in this debate. First Marc Joffe argued for the use of XBRL in municipal reporting. Then Friedrich Lindenberg responded that financial reporting should instead look to transport data in order to reduce complexity. Below follows the response from Charles Hoffman: + +XBRL has unfortunately earned the reputation it has because of (a) flaws in the way some regulators implement XBRL and (b) misunderstands of the business people promoting XBRL. This is very consistent with what Gartner calls the “hype cycle”. + +The following are the realities and truths which should be considered summarised as succinctly as possible. You can see the details here on my blog. + +### Point 1: Achieving meaningful exchange +“The only way a meaningful exchange of information can occur is the prior existence of agreed upon semantics, syntax, and workflow/process rules.” This video made available by HL7 explains this in more detail. + +###Point 2: Formality +If you consider point 1, the “rules” can be somewhat of a bottomless pit. A balance needs to be achieved between practicality (something actually works) and “formality” (spending so much time creating rules and making things so complex that no one could ever use the system). A practical balance needs to be achieved. + +###Point 3: Expressiveness +While it is true that CSV has been around a long time, it is easy to use, there is lots of support….CSV is not very expressive. CSV is a “flat” tabular structure, two dimensional. Information is “n” dimensional (could have many dimensions). An OWL ontology is WAY, WAY more expressive in terms of creating rules to make sure the information is correct (i.e. Point 1), but it is much more complicated because of that expressiveness. + +###Point 4: Complexity +While “complexity” can never be removed from a system, the complexity CAN be moved. What I mean by this is that while it is hard to create something like an OWL ontology, computer software can shield business users from the complexity in many, many different ways. One example is the use of “patterns”. Another is using “application profiles”. Another is using the 80/20 rule in terms of creating business rules to assure information quality. I could go on and on about this and show you many, many examples. Fundamentally this all boils down to the this one fact: “XBRL software vendors” are building the wrong software; they have built XBRL technical syntax editors instead of “digital financial reporting” applications or “digital business reporting” applications. This problem is understood by some software vendors who are now building the correct software, others are understanding, everyone will be forced to move in this direction due to market pressure. + +###Point 5: Guidance-based, semantic-oriented, model-driven, business report authoring enabled by “semantic web” technologies +Authoring business reports in the future will be as different as the difference between creating a photograph when you used a darkroom filled with smelly and chemicals as contrast to using “Photoshop”. What you can do with a business report will also be as different as what you can do with a photograph printed on a piece of paper and a photograph expressed digitally. The key is “metadata” and applications which understand and therefore leverage that metadata. For example, Microsoft Word knows ZERO about creating a financial report. Nothing. Guidance-based, semantic-oriented, model-driven financial report authoring tools (think TurboTax) will have: +• Knowledge baked in +• New knowledge can be inferred/added +• Agility to adapt to ever-changing conditions +• Semi-automated data integration +• Machine intelligence + +You may not be able to imagine these applications, or maybe you can. But when you see an application working correctly, leveraging a rich set of metadata (which you cannot even express using CSV files), it will be very, very easy to grasp these ideas. Read the documents linked do on this blog post. + +XBRL is only part of a much, much broader trend of digital business reporting and digital financial reporting. That is part of an even bigger trend, “digital”. Electronic medical records is an example of the much broader trend. Electronic medical records has many of the same issues as what the U.S. Securities and Exchange Commission (SEC) is trying to do with XBRL-based financial filings. The accounting profession and SEC is much, much further down the path than electronic medical records from what I can see. Electronic medical records (EMR) are not “interoperable” or exchangeable between systems yet (XBRL is). There is no international standard for EMR (there is for financial reporting, XBRL). + +Generally, people are having the wrong discussion! They are discussing syntax (i.e. CSV, JSON, XML, etc.) and they should be discussing “how the heck are we going to articulate and management semantics”. That is the discussion which needs to occur. This is very, very useful stuff. This is not about saying that CSV is bad and that XBRL is good. They are two different tools for different problems. Using the wrong tool to solve a problem is bad as well as inappropriately using a tool is bad! + +The goal as I see this is success. Success means (for business people) cost effective, easy to use, effective, robust, reliable, repeatable, predictable, scalable, secure (when necessary), auditable (when necessary), practical, business information exchange by business users between business systems. + +Below you will find a short video where Charles Hoffman explains XBRL: + + diff --git a/examples/openspending/content/blog/2013-09-18-governance-groups.md b/examples/openspending/content/blog/2013-09-18-governance-groups.md new file mode 100644 index 00000000..3bde8cbc --- /dev/null +++ b/examples/openspending/content/blog/2013-09-18-governance-groups.md @@ -0,0 +1,41 @@ +--- +redirect_from: /2013/09/governance-groups/ +title: Announcing the OpenSpending Steering Group +authors: +- Anders Pedersen +--- +OpenSpending's code base, datasets, and community have grown rapidly over the past years. In July we announced a proposal for a new community-based governance structure to support the continued growth of the project. + +The new **steering group** and associated teams of OpenSpending will provide that structure. These groups will oversee the development of the OpenSpending project and ensure broad community influence in future plans for the project. + +While OpenSpending has always been a community project, the new groups will reflect the formal recognition of a broad community of stakeholders. The legal, institutional, and infrastructural home of OpenSpending will remain with the Open Knowledge Foundation, and the Open Knowledge Foundation continues to be strongly committed to the project. + +## The Steering Group + +The new Steering Group will have overall responsibility for the direction and management of the OpenSpending project. The steering group is to work with the community to plan the overall strategy and to implement community decisions. The responsibilities also include managing the OpenSpending brand and the use of the OpenSpending name and setting and enforcing data licensing and other policies. + +Contributors from the OpenSpending community can put themselves forward to serve a two-year term on the Steering Group. Members of the existing steering group will review applications and appoint new members. To bootstrap the process, the Open Knowledge Foundation has invited members to the initial steering group from major OpenSpending stakeholders. + +The steering group will include: + +* Justin Arenstein, African Media Initiate (Kenya) +* Jay Bhalla, Open Institute (Kenya) +* David Hall-Matthews, Publish What You Fund (United Kingdom) +* Kaitlin Devine, Sunlight Foundation (United States) +* Elena Mondo, International Budget Partnership (United States) +* Oluseun Onigbinde, BudgIT (Nigeria) +* Simon Parrish, Development Initiatives (United Kingdom) +* Anders Pedersen, Open Knowledge Foundation (United Kingdom) +* Federico Ramírez, Fundar (Mexico) +* Adam Stiles, Open Budget Oakland (United States) + +## Teams + +Our News and Website team will run our community site, including the blog and resource pages. Members will also write stories connecting spending news with the data behind them. The team already manages the blog and conducts task management using this Trello board, which includes contributors from across the world. + +The data team oversees the contribution of new datasets, issues of data quality and privacy, and the management of the database. They assist community members in contributing new data and make decisions on data-related policies, standards, and legal issues. + +The Technical Team comprises the official committers to the OpenSpending code repository. They oversee the technical work, including all code contributions, and are responsible for maintaining technical infrastructure. + +Membership in the teams is determined by contribution. Existing team members recognise consistent good work by extending the invitation to join the team. + diff --git a/examples/openspending/content/blog/2013-09-18-partnership-with-cern-results-in-new-load-api.md b/examples/openspending/content/blog/2013-09-18-partnership-with-cern-results-in-new-load-api.md new file mode 100644 index 00000000..1d79bf44 --- /dev/null +++ b/examples/openspending/content/blog/2013-09-18-partnership-with-cern-results-in-new-load-api.md @@ -0,0 +1,51 @@ +--- +redirect_from: /2013/09/partnership-with-cern-results-in-new-load-api/ +title: Partnership with CERN results in new Load API +authors: +- albertorodriguezpeon +--- +This summer OpenSpending and the Openlab at CERN partnered on an internship programme to enhance financial transparency at CERN. Under the partnership Alberto Rodriguez Peon spent 8 weeks working with the OpenSpending developer community and supervisors at CERN to build the first load API for the tool. At OKCon Jiri Kuncar (CERN) who supervised the project will give a talk about the project at the session Open Finance and OpenSpending Wednesday 18 September, 14:45 – 16:00 @ Room 7, Floor 2. + +In this guest post Alberto Rodriguez Peon describes how he created the tool. + +### Problem +The collaboration between CERN and the Open Knowledge Foundation implies the need of sending financial data to OpenSpending, a project for mapping the financial transactions of governments and institutions. + +OpenSpending manages the input of new datasets through a CSV file with raw data and a form (containing information like name, country, language, currency, etc.) filled in manually by the user. + +However there is no opportunity for dynamically adding a dataset, which is what CERN needs in order to push financial information in an automated way. The OpenSpending API does not cover the process of introducing data, just searching and visualising it. To overpass this issue, we are developing this API for the OpenSpending site. + +### Solution +The idea consists of adding a new method to the existing API to complement the manual input of the metadata. + +For that, we replace the form with a JSON file containing all the information that we have to provide in order to create a dataset. This works in the same way that the internal tool “ostool” is used in OpenSpending for the installation and setup. + +So, technically, the API request should be something like this: + +

    POST /api/new?csv_file=&metadata= +
    +This information would be enough to process a dataset and add it to OpenSpending except that there is no way to know which user has actually made the request and therefore we do not know the creator of the dataset. + +Each OpenSpending user has an API key which can be used to identify himself for API request. + +
    POST /api/new?csv_file=&metadata=&apikey= +
    +The problem is that we cannot just put the API key in the request as anybody can intercept it and use it as if it was its own. + +### Authentication via API key +To solve this issue, we propose a solution using symmetric key as the authorization method (in an Amazon-like way). + +Instead of having only a public API key for each user, we create a “secret” one as well. The idea is therefore to put the public one in the request and adding a signature, calculated using the ‘secret’ API key and a cryptographic hash algorithm (in our case MD5). + +
    POST /api/new?csv_file=&metadata=&apikey=&signature= +
    +The signature is calculated concatenating all the params in the request, sorted alphabetically and starting with the ‘secret’ key. + +
    api_keycsv_filemetadata +
    +The generated string is ‘hashed’ with MD5 to obtain the signature. + +Therefore, in order to validate the user, the server will calculate the signature from the params and compare it against the provided one. If both are equal, the user is authenticated. + +The developed API will provide a tool for governments and organisations (including CERN) to publish data in OpenSpending without the heavy task of adding the data manually. + diff --git a/examples/openspending/content/blog/2013-09-24-features-in-july-and-august-2013.md b/examples/openspending/content/blog/2013-09-24-features-in-july-and-august-2013.md new file mode 100644 index 00000000..9a77e866 --- /dev/null +++ b/examples/openspending/content/blog/2013-09-24-features-in-july-and-august-2013.md @@ -0,0 +1,84 @@ +--- +redirect_from: /2013/09/features-in-july-and-august-2013/ +title: Features in July and August 2013 +authors: +- Tryggvi Björgvinsson +--- +We owe you a lot of development updates. Both July and August (and it's well past middle of September now). September is shaping up to be an awesome month, but more on that later (since there is still room for more awesomeness). In this post we'll be sticking to the highlights of development for July and August. + +![Image by Alan Cleaver (cc-by 2.0)](http://farm4.staticflickr.com/3071/2661425133_1328692483_z.jpg "Time span") + +### Front Page Changes + +Let's start with one of the reasons it took so long to get write this blog post. In July and August we started the work of migrating our blog from Jekyll to WordPress. Most of you have probably noticed this but it meant that we had two blogs running at the same time. Our main blog which ran at [community.openspending.org](http://community.openspending.org) (at the time it was blog.openspending.org) and then we had another, outdated version running on our front page. + +This setup was slightly embarrasing so we had to fix it and we did. We changed our front page so that it now shows *calls to actions* instead of showing a blog since we felt that this would better represent what people visiting [OpenSpending.org](http://openspending.org) would be looking for. + +### Dataset Page + +The dual-blog confusion isn't the only embarrasing thing about OpenSpending. The number of datasets in OpenSpending has grown spectacularly in the last few months and an irritating bug popped up. We aren't caching the datasets page properly which means that it takes *ages* (more like around 15 second) to generate a list of all datasets. + +This meant that when you clicked on the dataset link in the navigation bar nothing happened for about 15 seconds, and to most if not all people that looked like OpenSpending was broken. What was really happening was that in the background we were generating the list of datasets, but nothing seemed to be happening since this was all being done via AJAX (in the background with no status updates to the browser). + +We got a fix contributed that bought us some time where instead of doing everything in the background via AJAX, the link in the navigation bar now takes you to a web page with all of the datasets. It still takes about 15 seconds to generate the dataset but at least the browser gives users feedback (that it's still loading the page). + +This isn't the best solution. For example, an remaining issue is that the map on the front page which is generated with the AJAX thing doesn't show for 15 seconds. We really should fix the dataset list generation issue. It all boils down to doing caching properly. + +**Would you like to help us out?** All help with this issue is appreciated! We need to take a hard look at how we do caching and improve that to make OpenSpending faster and better for you, the users. You can offer your help as both a code reviewer or a coder. We can help you get started. Just let us know you're interested on our [developer mailing list](http://lists.okfn.org/mailman/listinfo/openspending-dev)! + +### Overwrite Drilldown Behaviour in Treemaps + +July and August haven't only revolved around embarrasing things for OpenSpending. There were a lot of awesome things that got implemented and deployed in July and August as well. One of those things was the possibility to overwrite drilldown behaviour in treemaps. Yes this is a complicated description but it's a really important and cool feature. + +[OpenBudgetOakland](http://openbudgetoakland.org/) is an awesome satellite site powered by data in OpenSpending and our visualisations (and mixes it with more interesting stuff in a cool way). The team behind the site wanted to give their users the possibility to comment on and share specific drilldowns in the treemap, and give their users some breadcrumbs so they could get back to previous levels. + +For this they needed to change the treemap so they could overwrite the default drilldown behaviour. This change has now found its way back upstream and is part of OpenSpending. So all you satellite site maintainers out there! You can now do the same thing as OpenBudgetOakland! + +If you want to add breadcrumbs and share locations you can do it in the same way as OpenBudgetOakland. The magic happens in [lines 47 to 66 of their main javascript file](https://github.com/adstiles/openbudgetoakland/blob/gh-pages/javascripts/main3.js#L47-L66) where they set the *drilldown* context variable to a self-defined function. You will also have to look at the preceding lines which generate the breadcrumbs based on the urls (since the magic is actually just about redirecting users to another page when drilling down into the treemap). + +### Public Profile Page + +Users of OpenSpending also now have a public profile page on [openspending.org](http://openspending.org). On the profile page you can now add your twitter handle and choose whether or not you want your email and/or your twitter handle to be visible to the general public (not only OpenSpending administrators). + +This makes it far more easier to get in touch with users who are managing interesting datasets (for example if somebody wants to help or meet up). User identity is also more visual since we now show the user's [Gravatar](http://gravatar.com), based on the user's email address. + +We also did some minor changes to the navigation bar so that you can see your identity more clearly and the navigation bar isn't as cluttered. Click on your username to access either your dashboard or your settings. + +After you log in you're taken to your new dashboard (profile page) and besides the old information which was shown in the old dashboard, and the email and/or twitter handle, we now also show a *Import a dataset* button so you don't have to go through those 15 seconds of loading a dataset list just to upload a new dataset (we need your help to fix it!) + +Speaking of help. There is a hidden embarrasing *bug* in the new navigation bar associated with the new profile page. We have since this spring been migrating from [Genshi templates](http://genshi.edgewall.org/) to [Jinja2 templates](http://jinja.pocoo.org/). We're not done yet and the changes to the navigation bar only affect the Jinja2 templates. So sometimes on OpenSpending you'll see the old navigation bar. We really need to finish the migration. There's really not much left to do. + +**Do you have any html skills?** If so, you can help us finish the migration from Genshi to Jinja2 so that it'll be more easier to work with OpenSpending's templates *and* users won't be presented with two versions of the navigation bar, depending on where they are on the site. + +### Data Range + +Are you interested in a particular dataset? Do you want to know if it gets updated? Maybe you're looking at a dataset and want to quickly see the time period for the data or when it was last updated? + +OpenSpending now shows important dates for a dataset. When looking at the entries for a dataset you can see at the bottom the time span for all entries (date of the earliest entry and the date of the latest entry). + +Also on the about page for the dataset you can now see the date when it was created and the date when it was last updated. Note that these are dates recorded for when things are worked on in OpenSpending, the entries time span is for the dates in the dataset itself. Two different things, but two important things when you're interested in some spending data. + +### Dailybread in Satellite Templates + +Keen readers might be thinking: "Wait, hasn't dailybread been in the satellite template since the beginning". The answer is: "Yes". However the version which has been in the template since the beginning was kind of useless. It really only worked if you wanted to build exactly the same dailybread as was used on [Where Does My Money Go](http://wheredoesmymoneygo.org/) so that doesn't sound much like a template. + +The big change is that the satellite template can now support a non-UK dailybread visualisation. The options of the dailybread visualisation can be overwritten and used in the template. Hopefully this will lead to many more interesting satellite sites. + +### Taxman goes around + +The dailybread fixes to the satellite template mean little to nothing if [Taxman](http://github.com/openspending/taxman/) isn't updated. So if you want to create a dailybread visualisation for your country you still need to add the tax calculations. In July/August we saw two new countries added to Taxman. *Bosnia and Herzegovina* (with all three of their entities which are taxed differently) and *Japan* (which also introduced a better quality assurance, build process and tests, to Taxman. + +Taxman is now up to 6 jurisdictions so there's still a lot of work left to do there. Is your jurisdiction missing? **You can add it to Taxman**. Just follow one of the six jurisdictions to see how it's done. It's easy! + +### Other Changes + +As you can see there were a lot of big changes to OpenSpending in July and August. There were many more changes, like for example the awesome OpenSpending icons page which is available at http://jmblog.github.io/openspending-icons/ (also linked from the OpenSpendingJS README). Some deprecated functionality was replaced in OpenSpendingJS meaning that we now conform to more recent versions of libraries we use. + +Remember that you can help develop OpenSpending. This is a community project and the software and all of its eco-system is only as strong as we are together. + +### Thanks + +Thanks to **Neil Ashton**, **Vitor Baptista**, **Michael Bauer**, **Tony Hirst**, **Yoshihide Jimbo**, **Andy Lulham**, **Prakash Neupane**, **Anders Pedersen**, **Adam Stiles**, **Marco Voormolen**, and **Kenan Zahirovic** for their contributions these two month (there are probably a lot more who've contributed somehow to OpenSpending so don't be sad if we forgot you - it's a bit hard to manage, just let us know and we'll add you). + +Image of clocks used for this blog posts is by [Alan Cleaver](http://www.flickr.com/photos/alancleaver/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). + diff --git a/examples/openspending/content/blog/2013-10-01-sevilla-presus13.md b/examples/openspending/content/blog/2013-10-01-sevilla-presus13.md new file mode 100644 index 00000000..ef1798b8 --- /dev/null +++ b/examples/openspending/content/blog/2013-10-01-sevilla-presus13.md @@ -0,0 +1,16 @@ +--- +redirect_from: /2013/10/sevilla-presus13/ +title: 'Sevilla Presus: Data-driven journalism at municipal level' +authors: +- J. Félix Ontañón +--- +A such a disruptive technology as Internet, is forcing us to re-think the role and methods of many professions. Old models hasn't died yet, but in the new ones we can find some common patterns for success: empowering people through a community for cooperation. Journalism isn't the exception. Through the The Guardian Data Blog, as an example, many citizens helped to transcribe and find stories in the MP's expenses data: people give their eyes, The Guardian gives the platform. + +Sevilla Actualidad and Sevilla Report, both local newspapers in Sevilla (Spain), are using two OKFN tools empower their fellow citizens in the spirit of The Guardian Data Blog. With the help of OpenKratio, a group of citizens for fostering the Open Government Data culture in Spain, we've launched #SevillaPresus13, a Crowdcrafting app to crowdsource the transcription of municipal budgets. + +#SevillaPresus13 + +We aim to complete a set of visualizations on OpenSpending for the 2011-2013 municipal budget series (2012 was done), so it would be available for everyone to embed them in their own web posts. The plan is to link the budgetary information with the municipal public procurement. This way both local newspaper will have a powerfull tool to monitorize the municipal activity and finding insteresting stories to tell. This project has been accepted into a data-journalism contest in Madrid (Spain). + +Crowdcrafting is an amazing platform to build crowdsourcing apps for transcribing document and images into machine-readable data. As it provides some out-of-the-box pdf transcribe apps, all you need to do is download, customize and deploy for your own proposes. In the case of PDF files, tools as Tabula are improving the way non-techie people can unlock the information, but only Crowdcrafting is able to develop an engaging crowd-experience for users. + diff --git a/examples/openspending/content/blog/2013-10-02-announcing-the-news-editors-and-data-wranglers-teams.md b/examples/openspending/content/blog/2013-10-02-announcing-the-news-editors-and-data-wranglers-teams.md new file mode 100644 index 00000000..9908ef7c --- /dev/null +++ b/examples/openspending/content/blog/2013-10-02-announcing-the-news-editors-and-data-wranglers-teams.md @@ -0,0 +1,164 @@ +--- +redirect_from: /2013/10/announcing-the-news-editors-and-data-wranglers-teams/ +title: Announcing the News Editors' and Data Wranglers' Teams +authors: +- Teodora Beleaga +--- +At OpenSpending, our restless community drives our work and so our volunteers are key to our activities. Today we are welcoming those joining the News Editors' and Data Wranglers' teams. Find out more about them and what they stand for in the short bios below. Many have already been at work, and you can follow their involvement on our Trello boards. + +We will kick-off tonight (lunchtime or breakfast depending on your timezone) with an experiment, a live-blog of the first joint Hangout between the two teams. Tune in at 6pm GMT (or thereabouts). + +

    News Editors

    +

    +

    These guys run the blog and manage our social media presence. We are still recruiting News Editor volunteers so if you'd like to join the team apply here. + +

    Burite Joseph, @BuriteJoseph + +

    Independent media practitioner and entrepreneur with over five years of journalism and research experience, Burite runs ZHENOBIA, a media integration and multimedia content aggregation company. She also consults for SMS Media Uganda, Ultimate Media Uganda, East African Business Week and Daily Monitor. + +

    +

    Working with data is my new passion. I am a quick learner and teamwork is my steroid. + +

    +

    Anna Flagg, www.annaflagg.com + +

    Data journalist at the Center for Responsive Politics, Anna has a background in computer science, data visualization, design and data-storytelling. + +

    +

    I like working on projects that create awareness of issues important to the public. I'm excited to work with and learn from the Open Spending community. + +

    +

    Laura S. García,@laura_s_garcia + +

    An experienced journalist, Laura has worked for more than ten years as a multi-media journalist in Spain. She has also taught Geography and History to high-school students. Laura speaks Spanish, Galician, English and a little Swedish. + +

    +

    I’m looking to improve my knowledge of open data, as I’ve always thought this to be the best way to offer a good journalism and a good education as well. + +

    +

    Karen Brzezinska, @westofwarsaw + +Also a professional journalist, Karen (Kati) worked for international news services specialising in equity, commodity and currency markets. Her background is in PoliSci (East European studies), and, while originally from midwestern US, her life experience lists Italy, Hungary (1989-1992) and The Netherlands (since 1992) as home-countries. Kati is fluent in English (US) and Dutch. + +

    +

    I'm interested in learning how open data can be used to enhance governance and education. + +

    +

    Dominic Kornu, @qaphui + +

    An IT and Maths tutor from Ghana, with an interest in web and social media technologies, Dominic blogs at Qaphui’s Cafe and volunteers in his free time. + +

    +

    I am interested in learning how open data can be used to enhance governance and education. + +

    +

    Mehmet Koksal, @mehmetkoksal + +

    Freelance journalist based in Brussels (Belgium) and conference interpreter, Mehmet also works as a fixer for the international press, including the French weekly Courrier int.. In his free time he volunteers for AJP and acts as a campaign manager for the EFJ. + +

    Teodora Beleaga, @t30d0ra + +

    A digital analyst and freelance data journalist based in London, Teodora is an alumna of City University’s Interactive Journalism MA and has completed work experience with The Guardian. + +

    +

    I joined the Open Spending project to share my data analysis skills and expand my understanding of fiscal transparency and government spending. + +

    +

    Maria Laverghetta + +

    A freelance journalist and editor from Orlando (Florida), Maria works for Pearson Education and theDailySource.org. A graduate of the University of Florida’s College of Journalism and Communications as well as the College of Music, Maria’s work focuses primarily on feature writing, editing and music. + +

    +

    I am continually trying to broaden my knowledge in this sector, more specifically at this juncture in finance and online education cultivation. + +

    +

    Miriam Ruhenstroth + +

    A Science and technology freelance journalist based in Berlin (Germany), Miriam has a background in biological sciences. In 2011 she attended a summer school for data journalism (organized by Initiative Wissenschaftsjournalismus). + +

    +

    I found the field of data storytelling thrilling and joined OpenSpending, to learn more about it and participate for good. + +

    +

    Data Wranglers

    +The Data Wranglers work to add, clean and visualise data in OpenSpending. They help community members who need assistance. Some data wranglers focus on cleaning and analysing data whereas others work to visualise data using the OpenSpending API. We are still recruiting Data Wrangler volunteers so if you'd like to join the team apply here. + +

    Concha Catalan, @conchacatalan + +

    An English teacher and freelance journalist based in Barcelona (Spain), Concha is currently working on a project to open the autonomous government of Catalonia (opengov.cat). She also blogs at http://barcelonalittleshell.blogspot.com.es. + +

    +

    I would like to add the data set of the autonomous government of Catalonia budget to OpenSpending. I am coming to terms with lots of new concepts. + +

    +

    Prakash Neupane, @nprkshn + +

    OKFN Ambassador in Nepal and FOSS Enthusiastic, Prakash is working in social development empowering individuals and communities by using technology. He is an Open Data Researcher and Nepali Wikimedian, responsible for Wikimedia Education Program in Nepal. Find out more  about him here. + +

    Pierre Chrzanowski, @piezanowski + +

    A member of the French OKFN working in the field of Open Government Data, Pierre says he is really interested to work on Tax Heaven, Public Procurement and Aid Data. + +

    +

    I want to learn more about tools to analyse the data sets and how best to do storytelling. + +

    +

    Samuel S. Lee, @OpenNotion + +

    Currently based in Washington DC, Samuel is a member of the World Bank Group Open Finances team. He loves data, innovation, transparency, photography and college football. + +

    +

    I am passionate about “open” and its potential to transform civic engagement, international development, and the world. I am particularly interested in realizing the potential of open financial information. + +

    +

    Adriana Homolova + +

    A data journalism student with a passion for open culture, Adriana is a member of the Society for Open Information Technologies. + +

    Sipos Zoltán + +

    A Hungarian journalist working for an Internet news portal in Romania, Sipos specializes in investigative reporting.  His background includes philosophy, sociology and public policies. Sipos has experience working with data, filing FOI requests, and tackling spreadsheets. + +

    +

    I am trying to learn as much as I can about data journalism through online groups, MOOCs and books purchased from Amazon. My ultimate goal is to set up a small investigative / data journalism start-up in Romania. + +

    +

    Gabe Sawhney + +

    A member of Better Budget Toronto Gabe joined the Team to bring transparency to his city’s budget. + +

    +

    I want to mobilize action (citizens, elected officials and policymakers) for better process, better clarity, better formats, and more transparency around city budgets. + +

    +

    Elaine Ayo, @eieayo + +

    Statistician student based in Washington, DC Elaine has spent the last three years in Seoul, South Korea as a copy editor for an English news wire. Prior to that Elaine reported for her hometown paper, the San Antonio Express-News, in Texas. + +

    Alessandro Martinisi + +

    A PhD student in Journalism and Mass Communication at the University of Westminster, London, Alessandro shares an interest in Data-Driven Journalism. He has previously worked in South America (Brazil and Argentina) for a couple of years for the communication unit of the United Nations, UNPD, as a journalist and documentaries writer. He says it was a landmark experience. + +

    +

    I started this new pathway in January/February and soon I started to keep myself busy trying to understand the new journalistic practice in which all of us are engaged: Data-Driven Journalism. + +

    +

    Hans Loos + +

    An IT and telecom freelance journalist based in Belgium, Hans studied sociology and has a passion for statistics. + +

    +

    I have started to learn to program and study R but without big results up till now. + +

    +

    Rochelle Sampy + +

    Freelance journalist based in the UK, Rochelle is currently working for a B2B pharmaceutical publication. WIth a background in Law she previously worked with the Centre of Investigative Journalism where she first discovered data cleaning. + +

    +

    I would like to learn more about data wrangling in order to better my knowledge of its use for investigative journalism. + +

    +

    +  + diff --git a/examples/openspending/content/blog/2013-10-02-news-editors-and-data-wranglers-hangout-live.md b/examples/openspending/content/blog/2013-10-02-news-editors-and-data-wranglers-hangout-live.md new file mode 100644 index 00000000..f4a0ae6b --- /dev/null +++ b/examples/openspending/content/blog/2013-10-02-news-editors-and-data-wranglers-hangout-live.md @@ -0,0 +1,26 @@ +--- +redirect_from: /2013/10/news-editors-and-data-wranglers-hangout-live/ +title: News Editors and Data Wranglers Hangout - Live +authors: +- Teodora Beleaga +--- +The News Editors' and Data Wranglers' Teams are catching up tonight in a joint Hangout. In the interest of transparency we are running this live-blog as an experiment. + +Refresh at your convenience. + +7.02pm (GMT) We are wrapping up, as our to-do lists are growing. Final points include allocation of more administrative tasks and an awesome project that Steve came up with which is to map all monies spent on energy management, specifically with regards to global warming and the funds budgeted there. + +6.49pm (GMT) Berlin-based Michael is sharing his project of opening up budgets across German cities, including this one and this one. In partnership with Miriam Ruhenstroth they aim to get the data up on Open Spending. + +6.42pm (GMT) Another insight from the call is that our guide, which aims to help communities get started and has been translated (by volunteers) in four languages, may yet again be in need of a refresh. Steve is going to go through it with a pair of fresh eyes and help update it. If you want to help too, please let us know. Getting communities started as seamlessly as possible with Open Spending is very important to us. + +6.32pm (GMT) We're talking data cleaning now. Big Data that is. Elain's data is a 1GB file of over 1m lines. She is currently attempting the task with MySQL and OpenRefine. Do you have any other suggestions? + +6.27pm (GMT) One of the data wranglers, Elaine Ayo is working cleaning the data on EU structural funds for Italy, which she claimed here. "It's quite large and so I am still working on trying to figure out how to deal with it," she told us. Can you help? + +6.22pm (GMT) Turns out we have Berlin and Toronto represented in the call too. + +6.17pm (GMT) We are taking rounds to reinforce who we are and what we're in this game for. George Adcock (aka Steve) underlined the need for an infographic map showing the relationships between all the tools we use, especially here at OpenSpending, whether to produce our work, keep up with it or share it. Great insight! + +6.10pm (GMT) Welcome! Hooking up on a call from across the world take a few minutes. We are joined from DC, London and Louisiana to name but a few places. In the call tonight we have Anders Pedersen, George Adcock, Elaine Ayo, Michael Horz, Neil Ashton and myself. + diff --git a/examples/openspending/content/blog/2013-10-04-features-in-september-2013.md b/examples/openspending/content/blog/2013-10-04-features-in-september-2013.md new file mode 100644 index 00000000..0adfa253 --- /dev/null +++ b/examples/openspending/content/blog/2013-10-04-features-in-september-2013.md @@ -0,0 +1,104 @@ +--- +redirect_from: /2013/10/features-in-september-2013/ +title: Features in September 2013 +authors: +- Tryggvi Björgvinsson +--- +![Image by NCTRUCKINGITEMS (cc-by 2.0)](http://farm6.staticflickr.com/5305/5865630148_7c430641dc_z.jpg "Loading...") + +Like we said in our last development update, September was going to be awesome. And boy oh boy did that come true. There were a lot of great features added to OpenSpending. Some of them we can safely say might change the way people use OpenSpending. + +### Multiple measure aggregation + +The *amount* measure is required in all datasets in OpenSpending but that does not mean that it has to be the only measure. You can add how many measures you like. For example, amount could show the amount actually spent and you could have another measure *budget* to show what was budgeted. + +For some time now it has been possible to use another measure than amount when aggregating via the api. The only thing you'd have to do is add a url parameter *measure*, for instance *measure=budget* to aggregate over the budget measure instead of the amount measure (amount is default). + +However there was a problem if you wanted to show the budget *and* the amount on the same page. You would have to make two requests to OpenSpending, one for the budget and another for the amount. This would make your site (or whatever you have that accesses the aggregation API) slower and adds extra processing to OpenSpending (since it now has to access this twice). + +We have solved this! Now you can aggregate multiple measures in the same request. You still use the same url parameter (*measure*) and you separate the measures with a | just like you do with drilldowns: *measure=budget|amount* + +### Broken terms of use + +Sander van der Waal noticed something really bad about OpenSpending. When users register for an account on OpenSpending they have to agree to our terms of use. There's a link to the terms of use on the site but that link was broken and has been broken for some time. + +Does that tell us more about how many people actually read terms of use or has this been driving people away from registering? We can't know but if it is the latter then this has been fixed now. You can now actually access the terms of use and read them. So if you didn't want to because you couldn't see the terms of use but agree to them now that you can read them, please come and register. + +### Timestamps in dataset.json + +In August we added important dates to web pages for the datasets so that users can quickly see when a dataset was created or last updated and the time range of the dataset. + +You couldn't access this from the API though. Well you can access the timestamps in the dataset by fetching the JSON at http://openspending.org/**a-dataset-id**/time.distinct.json and build the date range. But the timestamps for the dataset in OpenSpending, i.e. created and last modified, were not accessible via the API. + +We have now added them to the dataset's json representation. So if you fetch the json at http://openspending.org/**a-dataset-id**.json you'll see an object called timestamps where you can access the created and last_modified timestamps. + +### Cleanup of issue tracker + +At our September developer meeting we discussed our issue tracker which had passed 100 open issues. We decided to move a few of them into OpenSpendingJS and take a stab at cleaning up any remaining issues. Mark Brough from [publishwhatyoufund.org](http://publishwhatyoufund.org) stepped up and helped us get the issue tracker into shape. We're now down to roughly 60 issues and we've tried to label them (there are still some unlabelled issues left). + +Mark Brough actually suggested we should set ourselves a goal of having less than 50 (or even 30) open issues at a given time. That's a great goal and we should definitely aim for it. + +**Do you want to help us achieve it?** Don't know where to start? We've labelled issues with *Volunteer: simple*, *Volunteer: medium* and *Volunteer: hard* in our [issue tracker](http://github.com/openspending/openspending/issues). Take a look at [how we hack on OpenSpending](http://community.openspending.org/help/development/volunteer) and start submitting pull requests. It's very easy and we're a very friendly newcomer community! + +### Source control + +When adding data to a dataset users add a source, analyse it, test it and then load it. The sources added can sometimes have errors in them, errors that are discovered in the analysis, testing, or the loading phase. This means that the source editor can get pretty cluttered. + +We now added a possibility to do some source house cleaning. You can remove sources that haven't been successfully loaded into OpenSpending. The reason you can't remove a loaded source is that we can't be sure if we'll be able to access the source url again to find the right entries (watch out for our data package support which might solve this). + +We also tried to improve the user interface a bit. You usually could only work with the source you added last. We now allow you to work with (and therefore delete) older sources... and we have **shiny buttons**! + +### Inflation adjustments + +A lot of users want to be able to compare datasets between years. How much is my government spending on health care this year compared to 5 years ago? This is a common use case but if one looks closely, all across the world governments seem to be spending more on everything than the previous years (well... it depends). The point is, because of inflation everything goes up and skews the numbers. You can't really compare data from now with data from 5 years ago unless you adjust for inflation. After taking inflation into account you might see that your government is actually spending less on health care than it seemed like. + +Since this is such a commons use case we felt it was quite important to support inflation adjustments in OpenSpending and now we do! Woah! This is huge! We can do it on an entry basis and for aggregations. The only thing required is for you to add an inflate URL parameter with the date you want to inflate everything against, *inflate=2012-04-22*, so that all amounts will be represented as they were valued at that particular date. + +Now, there's a huge catch to this. Inflation adjustment in OpenSpending is far from perfect. We don't have good enough data to do proper adjustments because data is released infrequently and valued differently across countries. The data is also quite granular. The date 2012-04-22 will actually inflate against 2012-01-01 because we only have annual inflation data and that's incorrect. We only get the year 2012 in the data we're using so we have no idea if it's January 1 or December 31 and that's a whole year. So for now we just do the same thing as OpenSpending. When we only have year as value we default to January 1 (even though it's probably more likely that the value applies to the end of the year). + +Since data is released a long time after it has been measured we can't even do realtime data. So you won't be able to inflate to the value of money you understand (the current value). You'll have to use data from last year and mentally try to remember how money was valued then, which is problematic -- do you remember the price of bread a year ago? + +But even if we cannot represent the value at the current price we can now compare multiple years correctly, by adjusting for inflation (unless the data is so recent we don't have inflation data for it). The remaining problems can be fixed since they just boil down to bad data being published. We need better data and we need it now! + +**We need your help!** We manage our CPI data, which we use to do inflation adjustment, in a special [data package repository](http://github.com/datasets/cpi). You can help us maintain it. Can you lobby for, find, and add better and more granular data for your country or other countries? If so you would not only help users of OpenSpending, but users of CPI data all over the world (who would like to use this data package). + +### Rename visualisations + +About a year ago, J. Félix Ontañón reported an issue where he wanted to be able to rename the visualisations on openspending.org. It took a while but we have now fixed this (sorry for taking so long fontanon). + +We actually did more than what was asked for. You can update the name of the visualisation, the description and the visualisation itself. So if you've added a new year to your dataset and the visualisation is for an older year, you can update it to show the most recent year. + +The only thing you have to think about is that there might be others that have embedded your visualisations in blog posts or web pages. Now you become a maintainer like us. With OpenSpending we always have to think about how changes affect users of the database and if web sites that rely on OpenSpending might stop working. You have to think about this as well: Will an update to your visualisation break pages that embed it? + +### Archive a dataset + +Another issue reported about a year ago got fixed as well. Miro Scibrany asked for a feature to make a copy of a complete dataset. This is now possible but only if you have system administrator access to the server (this uses the ostool command that comes with OpenSpending). + +You can use the same command *ostool* to load an archived dataset into OpenSpending so this is a really handy way to move complete datasets around. The problem is that it's not publicly available but we don't see this as something everyone wants to do since this mostly just helps people who have a development instance running get data from the central database without too much difficulty (and these people are probably already in touch with the core team that can get datasets for them). + +If in the future we ever decide to do federation of data we at least now have a working solution to do it which is exactly what the loading API was built around (a working solution). The loading API you ask? + +### Loading API + +Yes. The Loading API. The biggest improvement to OpenSpending this month came from [CERN](http://cern.ch) (it's so awesome to say that our loading API came from the same place as the world wide web). Alberto Rodriguez Peon did a great job with help from fukami and Jiri Kuncar to add a possibility to add and update datasets via the API. + +This means that you can now create a script that can scrape data from some spending sources you know and immediately upload it into OpenSpending without having to go through the OpenSpending web page (unless you desperately want to see the fancy buttons in the source controls). This is just awesome news for everyone who add data to OpenSpending. You can automate it! Just write a script, sit back, relax, and have a coffee (or something). + +Interested in the loading API? You can see how simple it is in [our documentation](http://community.openspending.org/help/api/loading/). + +One thing to note is that Alberto Rodriguez Peon developed a authentication mechanism for the API to prevent others from intercepting and gaining access as someone they're not (by getting their API key). This makes everything slightly complicated for the users (not too complicated). Currently this is the simplest way to do it while remaining secure but we've already decided how we're going to make it simpler but still have it secure so you can expect a simpler authentication pretty soon (there's only so much you can do each month) but the loading API itself won't change so you can start using it now. + +**If you write loading scripts let us know!** Let's not re-invent the wheel over and over again. If you've created a script make it free and open source so others will benefit (and others might then improve it for you when we get the simpler authentication). + +### Other Changes + +As always there were smaller changes made to OpenSpending in September (which was a month of huge improvements). We for example renamed blog.openspending.org to community.openspending.org, improved our documentation on the site, and much more. + +Remember that you can help develop OpenSpending. Can we top September? If you help us then we have a chance. It's going to be a challenge keeping this pace so come and help us out. Just have a look at our [OpenSpending issue tracker](http://github.com/openspending/openspending/issues) or our [OpenSpendingJS issue tracker](http://github.com/openspending/openspendingjs/issues). + +### Thanks + +Thanks to **Neil Ashton**, **Vitor Baptista**, **Michael Bauer**, **Mark Brough**, **fukami**, **Jiri Kuncar**, **J. Félix Ontañón**, **Anders Pedersen**, **Alberto Rodriguez Peon**, **Rufus Pollock**, **Miro Scibrany**, **Nick Stenning**, **Marco Voormolen**, and **Sander van der Waal** for their contributions (there are probably a lot more who've contributed somehow to OpenSpending so don't be sad if we forgot you - it's a bit hard to manage, just let us know and we'll add you). + +Image of men loading a truck used in this blog post is by [NCTRUCKINGITEMS](http://www.flickr.com/photos/oldtrucks/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). + diff --git a/examples/openspending/content/blog/2013-10-07-wearing-your-examples.md b/examples/openspending/content/blog/2013-10-07-wearing-your-examples.md new file mode 100644 index 00000000..98965393 --- /dev/null +++ b/examples/openspending/content/blog/2013-10-07-wearing-your-examples.md @@ -0,0 +1,58 @@ +--- +redirect_from: /2013/10/wearing-your-examples/ +title: Wearing your examples +authors: +- Tryggvi Björgvinsson +--- +![Spending t-shirts at Googleplex](http://farm6.staticflickr.com/5343/10051655705_a48d349b5c_z.jpg) + +[OpenSpending](http://openspending.org) and [Journalism++](http://jplusplus.org) are collaborating on a project we call *How much is it really?* The goal of the project is to create a webapp where users can type in a number to rough equivalents. + +The idea is that a reader of the news might be reading about how much it cost to built an airport. It might say it cost millions of dollars. The problem is that most news readers don't have millions of dollars nor will they ever have. They can't even begin to understand how much it is (but they do know that it's probably a hefty sum). This news reader could then quickly go and type in the cost mentioned in the news article and see that this cost half of what another airport cost or double what the government spent on education. + +We hope that this webapp will help readers of the news, journalists or just about anybody better understand the numbers or start questioning where money is going. Why does it cost so much to build airports? Why not settle for something less expensive and improve education instead? + +The airports and education example is fictional. I don't know how much it costs to build airports and I don't have in my head the amounts governments spend on education. But an example like this is the best way to explain the idea behind the webapp. + +Now OpenSpending and Journalism++ were invited to a meeting in Palo Alto by the funder of the project, [The Knight Foundation](http://knightfoundation.org) (the webapp is a subproject of a bigger project called *Spending Stories*). At the meeting we wanted to quickly give people an idea of what the webapp was all about. + +Instead of giving people a fictional example of airports and education we decided to use real examples and link them to an amount people understand and if there is an amount hackers and activists understand then that amount would be the price of t-shirts with witty things printed on them. So we decided to print t-shirts with real examples on them. + +But what to put on those t-shirts? There were three of us going to Palo Alto and each of us from a different country: Denmark (Anders), France (Anne-Lise), and Iceland (me). That's a great opportunity to give examples from all over te world (OK, not all over the world since we're all from Europe). We just needed to look at data we had for each of those countries in OpenSpending. Then we had to write those examples down and create some pretty images (or just use the ones we have in [OpenSpendingJS](http://jmblog.github.io/openspending-icons/) since they're awesome). + +## Danish version + +For Denmark we could look at all of the different municipalities and their spending. Made even easier by the ["Kend dine kommune"](http://kommune.politiken.dk/) project by the Politiken newspaper. After looking around tax revenue paid every year by an average inhabitant amounted to about 365 t-shirts. First example down! The resulting images and text: + +![Danish t-shirt version](http://farm6.staticflickr.com/5472/10051615784_cd29f67cea_z.jpg) + +Now here's one thing about that t-shirt. The image of the Little Mermaid (a monument for Copenhagen) is taken from [flaticon](http://www.flaticon.com/free-icon/little-mermaid-bronze-sculpture-in-denmark_852) and appears to be licensed under Creative Commons Attribution and the designer is said to be *Freepik*. + +The problem with that is that *"Freepik is a search engine that helps graphic and web designers to locate high quality photos, vectors, illustrations and PSD files for their creative projects."* It's not a designer. So I didn't want to attribute this incorrectly on a printed t-shirt (I probably should have just gone with a silhouette of Denmark but it's too late now). Anyways. See this as an attribution to Freepik but also me asking who the real designer is since I want to attribute the designer and not the search engine (and reimburse that designer for not attributing on the t-shirt). + +## French version + +France was slightly more difficult than Denmark. There was no good data in OpenSpending -- look at this as a call for action if you're from France! However Anne-Lise pointed out a recent scandal that had popped up recently surrounding [a charity website by Carla Bruni-Sarkozy](http://www.theguardian.com/culture/2013/jul/26/carla-bruni-sarkozy-foundation-website). The cost of this charity website was about €410,000 which is way more than you'd need to create a WordPress site. What's worse is that she paid for this site while she was France's first lady... and she paid for it with public money! + +Nicolas Bousquet (a web designer quoted in the news article) said anyone could have built this site for less than €10,000. With those amounts in hand we could mix them up with the estimated number of homeless children in France: + +![French t-shirt version](http://farm8.staticflickr.com/7294/10051656555_ab26986260_o.png) + +## Icelandic version + +Icelandic version was simpler since OpenSpending had data on [how much was spent my ministries in Iceland in 2012](http://openspending.org/uppgjor-rikissjods). These were probably the simplest calculations but the most interesting amount is probably how much the government spends on the welfare system since that's where it spends the most money (Nordic welfare system ftw!) even though our national hospital is almost bankrupt. Anyways looking at those numbers and how much Icelanders spent on average every second in the welfare system this was the result: + +![Icelandic t-shirt version](http://farm8.staticflickr.com/7324/10051762113_4932787e19_o.png) + +## Final t-shirts + +So how did the t-shirts come out. Well, me being the perfectionist I am not as I had hoped but they were still awesome. We were going to have colour versions with better prints but when the t-shirt printer called to let me know they wouldn't be able to print this because of details in the pictures I had to improvise a little. Since the graphics were all based on a specific price I had to go with normal prints on white t-shirts of better quality than the coloured t-shirts (got those for the same price). Nonetheless, the resulting t-shirts are still awesome: + +![Printed t-shirts](http://farm8.staticflickr.com/7422/10051762633_de55735d23_o.jpg) + +## Bonus + +It was really fun finding these examples and there were a lot of different ideas that came up. But since I mentioned airports in the initial fictional example I thought I'd throw in a t-shirt version that's connected to airports. It's an alternative French version (the looks aren't the same since this was scrapped before fixing fonts and adding logos): + +![Alternative French version](http://farm4.staticflickr.com/3785/10051687156_5e833d57e6_z.jpg) + diff --git a/examples/openspending/content/blog/2013-10-29-budzeti-ba-follow-the-money-bosnia-herzegovina.md b/examples/openspending/content/blog/2013-10-29-budzeti-ba-follow-the-money-bosnia-herzegovina.md new file mode 100644 index 00000000..3d9be765 --- /dev/null +++ b/examples/openspending/content/blog/2013-10-29-budzeti-ba-follow-the-money-bosnia-herzegovina.md @@ -0,0 +1,14 @@ +--- +redirect_from: /2013/10/budzeti-ba-follow-the-money-bosnia-herzegovina/ +title: 'budzeti.ba: following the money in Bosnia and Herzegovina' +authors: +- Neil Ashton +--- +budzeti.ba tax calculator + +We're pleased to announce the launch of [budzeti.ba](http://budzeti.ba), a new way to understand public spending in Bosnia and Herzegovina (BiH). budzeti.ba is a joint project of Bosnian civil society advocates [CPI](http://cpi.ba) and the [Open Knowledge Foundation](http://okfn.org). It allows Bosnian citizens to navigate the complexities of the BiH tax system and to explore the distribution of public funds across administrative regions and categories of social spending. + +budzeti.ba is designed to make public spending accessible to every Bosnian citizen. Users can inspect the details of national spending on categories like health and education through a simple point-and-click interface, and they can dig down into regional spending priorities by clicking through an interactive map. budzeti.ba's tax calculator feature allows Bosnians to learn how much they contribute to public services and where it goes by specifying their home region and their monthly income. No specialized knowledge is required to use budzeti.ba or to understand its intuitive presentation of spending. + +The budzeti.ba site brings together data from several sources, such as government websites and the Official Gazette of Bosnia and Herzegovina. The site’s initial launch is based on data from 2012 and will be updated as new data becomes available. CPI intends to deepen budzeti.ba's regional coverage by including municipal data and to enrich its analysis of government spending with data from the [Open Budget Survey](http://internationalbudget.org/what-we-do/open-budget-survey/), infographics on debt and deficit, and more. + diff --git a/examples/openspending/content/blog/2013-10-30-human-centered-data-takes-from-our-okcon-workshop.md b/examples/openspending/content/blog/2013-10-30-human-centered-data-takes-from-our-okcon-workshop.md new file mode 100644 index 00000000..98f22a0b --- /dev/null +++ b/examples/openspending/content/blog/2013-10-30-human-centered-data-takes-from-our-okcon-workshop.md @@ -0,0 +1,77 @@ +--- +redirect_from: /2013/10/human-centered-data-takes-from-our-okcon-workshop/ +title: 'Human Centered data: Takes from workshop at OKCon' +authors: +- Anders Pedersen +--- +The [Human Centered Data workshop][1] held at [OKCon 2013][2] by the [Sunlight Foundation][3]'s [Kaitlin Devine][4] and [Júlia Keseru](https://twitter.com/jkeserue) asked: what would public spending data look like if it were actually designed to be used? + +The workshop wisely refrained from looking for a single answer to that important question. Instead, it recognized that there are as many answers as there are types of users. It asked workshop participants to sort themselves into "basic" and "advanced" user categories and to explore what spending data designed for each group would look like. + +## Basic asks from spending data +Júlia Keseru headed up a discussion to address the needs for non-technical users of spending data who wish to access and utilise spending data: from citizens to journalists and activists. Several participants in the group emphasised the importance of having access to user friendly presentations of spending data and not just raw spending data in order to broaden access across levels of data literacy. + +The need for enabling citizens to comment and ask questions about actual spending items data was also highlighted. Though this is not possible today it could enable the citizenry to engage more directly on budgeting priorities they care about. + +### Unit costs of services +One of the issues often arising as a barrier when citizens or journalists attempt to engage with spending data is the lack of access to the actual unit costs of services. For example a purchase of school books for 1 mio. USD might appear justifiable at first glance, but ultimately it will depend on whether the unit price is 10 or 200 USD per school book. Today only few countries provide information about unit costs though the participants agreed that this could help the public gain a better understanding of the proportions behind the numbers. + +### Beware of digital divide +Some participants flagged the challenge of disseminating spending information to communities with little or no access to the internet. Radio was mentioned as a communication channel often under valued, along with community boards and other low tech solutions, which were also emphasised. Finally SMS was mentioned as option for distributing tailored spending information, which could be configured based on Q&A services. + +## Advanced users +The "advanced" group brought together participants from organisations like [Publish What You Fund][5] and [Development Initiatives][6]. It was conducted by Kaitlin Devine, who got the ball rolling by suggesting the broad outlines of the ideal spending data and allowing participants to flesh them out. + +The result was a portrait of spending data as designed for the technical leaders in financial transparency. Surprisingly, most of the discussion centered not around data itself but rather around the contextual information associated with it. + +## Documentation + +The first requirement for well-designed spending data identified by the power user group wasn't actually about data. Instead, it was about the data's *documentation*. Serious data-driven policy analysis, the group agreed, calls for good documentation of data's underlying methodology and domain. + +What needs to be documented is, in short, every non-trivial decision that goes into creating a dataset and choosing how to set its values. This includes the methodology behind the data, including the representation of ambiguous spending items such as loan guarantees: are they represented as "0", since no money is actually spent on them, or are they calculated as some kind of loss? The business rules underlying the data must also be identified (allowing users to avoid double-counting when aggregating data, for example) or else incorporated in the data in the form of structural links. + +Guidance on the expected contents of the dataset is also necessary. This includes information on data quality, both an assessment of the dataset's production quality and information on the normal values for different parameters in the dataset. Information on what is and is not contained in the dataset – for example, whether defense spending is included – is also necessary to make sense of the data. + +## Categorization + +The second need identified by the advanced data users didn't directly concern data, either. It highlighted the need for *metadata* that provides information about the dataset's semantics. + +The most important metadata concerns *categorization*, spelling out the classification scheme used to sort spending events into types. Many different such schemes are in use, and the group recognized the need for either a standardized international set of categories or for some sort of third-party "interlingua" to bridge the different schemes. + +Other important metadata needs include: + +* Granularity (agency, sub-agency, recipient, ...) +* Timestamps and "last updated" info +* Responsibility: who approved the spending in the dataset? Who signed off on it? +* The source of the spending data +* The step in the budget / signing / etc. process represented by the dataset (e.g. proposed, adopted, part-year completed, actual, ...) + +## Interoperability + +As you might expect from the fact that [a whole OKCon workshop][7] was devoted to the topic, making spending data interoperable was identified as a major need. In practice, this means establishing standards and consistency in identifiers and codes—or doing the work of building bridges between standards already in place. + +Good data should have consistent and unique identifiers for transactions, programs, and legal entities. That is, data values that refer to real-world entities and spending events should do so consistently within and between datasets. This is a minimum—the best case would be the existence of a standard format recognized by all data producers. In any case, the availability of machine-readable lookup tables for codes used in datasets is also important. + +Does establishing interoperability mean imposing a new standard for categories and identifiers, "one standard to rule them all", on all data producers? Not necessarily. Organizations like UNICEF say that standards should be adapted to local needs—governments who use whatever categories work best for their countries are in the right. The heavy lifting, UNICEF says, should be done by outside agencies, who should identify *mapping principles* that connect standards rather than prescribing new ones. + +## Community and context + +At this point in the workshop, one participant wondered if we hadn't missed the point. In concentrating on technical issues like metadata and identifiers, weren't the *human questions* at the heart of data getting lost? This question led the conversation to an interesting new topic: the need for means of identifying the *context* of datasets. + +How do budget and spending datasets relate to policies and political agendas? What is their meaning in terms of performance on some goal? These are the kinds of questions that we often ask of data, and we typically have our work cut out for us. This doesn't need to be the case: the information we want generally already exists but simply isn't linked to the data. Creating linkages between datasets and policy documents could address the first question, and establishing links between spending data and its stories of success (and failure)—perhaps along the lines of OpenSpending's [Spending Stories][8] project—could address the second. + +## Conclusion + +The workshop concluded with the conversation still underway, spilling over its time limit—data power users have lots to say about how data can better meet their needs! + +One thing is clear, however: data users want data that comes packaged with its interpretation, or rather the documentation and metadata necessary to reconstruct its meaning. Far from being transfixed by mere technical issues, data users are interested in the *meaning* of data, and they want to make it as easy as possible to gain access to that meaning. Data producers therefore need to worry less about producing data itself and start thinking more about producing data documentation and metadata. + +[1]: http://okcon.org/open-data-government-and-governance/session-4/ +[2]: http://okcon.org/ +[3]: http://sunlightfoundation.com +[4]: http://kaitlindevine.com +[5]: http://www.publishwhatyoufund.org/ +[6]: http://devinit.org/ +[7]: http://okcon.org/open-data-government-and-governance/session-g/ +[8]: http://blog.okfn.org/category/okf-projects/spending-stories/ + diff --git a/examples/openspending/content/blog/2013-11-01-joined-up-data-connecting-transparency-initiatives.md b/examples/openspending/content/blog/2013-11-01-joined-up-data-connecting-transparency-initiatives.md new file mode 100644 index 00000000..b7f193d7 --- /dev/null +++ b/examples/openspending/content/blog/2013-11-01-joined-up-data-connecting-transparency-initiatives.md @@ -0,0 +1,33 @@ +--- +redirect_from: /2013/11/joined-up-data-connecting-transparency-initiatives/ +title: 'Joined Up Data: first steps towards connecting transparency initiatives' +authors: +- Neil Ashton +--- + + +Unprecedented amounts of information on the financial resources available to fight poverty are now being released through the efforts of multi-stakeholder transparency initiatives in sectors like aid, construction, contracting, and extractives. Unleashing this data's potential will mean *joining it up*, following the money by merging and comparing datasets from different sources. Transparency initiatives have the opportunity to make this possible by identifying the shared *building blocks* of their data standards that enable comparisons + +[caption id="" align="aligncenter" width="640"] Photo credit: European Commission on DG ECHO[/caption] + +With this opportunity in mind, [Development Initiatives][1] (DevInit), the technical leads of the [International Aid Transparency Initiative][2] (IATI) and designers of the [IATI standard][3] for resource flow data, have initiated the **Joined Up Data** project. With this project, they aim to build a new culture of collaboration among multi-stakeholder transparency initiatives and to begin work on developing a shared set of building blocks for their data standards. + +Joined Up Data has begun with a [scoping study][4] developed for DevInit by the Open Knowledge Foundation and debuted at a workshop at the [Open Government Partnership][5] summit. The study identifies opportunities for collaboration between five transparency initiatives in different sectors: aid ([IATI][6]), construction ([Construction Sector Transparency Initiative][7]; CoST), contracting ([Open Contracting][8]), extractives ([Extractive Industries Transparency Initiative][9]; EITI), and fiscal transparency generally ([Global Initiative for Fiscal Transparency][10]). It assesses the coverage of each initiative's data collection requirements and identifies areas of overlap that could become fruitful sites of collaboration, and it discusses the implementation challenges faced by each initiative and the governance processes underlying the creation of each initiative's disclosure requirements. + +The scoping study demonstrates that there is much to discuss and a great deal of work to be done. The five initiatives covered in the study would all benefit from the development of building blocks covering organisational identifiers for government entities, geospatial data for sub-national administrative boundaries, and data standards for contracting information. The initiatives also have much to learn from one another's structure: bottom-up initiatives like EITI and CoST can share the view from "on the ground", whereas more top-down initiatives like IATI can contribute technical guidance. + +The Joined Up Data project is just taking its first steps, with future iterations promising to both deepen and broaden its coverage. The project also needs *you*. Improving the experience of using data is the most important goal of the Joined Up Data project. If you work with transparency initiative data (or want to), the organizers of Joined Up Data will surely want to hear how you use data and how it can be made more useful to you as they plan the next steps of the project. + +Access the scoping study Joined Up Data [PDF]. + +[1]: http://devinit.org +[2]: http://www.aidtransparency.net +[3]: http://iatistandard.org +[4]: http://devinit.org/report/joined-data-building-blocks-common-standards/ +[5]: http://www.opengovpartnership.org +[6]: http://www.aidtransparency.net +[7]: http://constructiontransparency.org +[8]: http://open-contracting.org +[9]: http://eiti.org +[10]: http://fiscaltransparency.net + diff --git a/examples/openspending/content/blog/2013-11-05-who-should-control-the-budget.md b/examples/openspending/content/blog/2013-11-05-who-should-control-the-budget.md new file mode 100644 index 00000000..fdd9888f --- /dev/null +++ b/examples/openspending/content/blog/2013-11-05-who-should-control-the-budget.md @@ -0,0 +1,18 @@ +--- +redirect_from: /2013/11/who-should-control-the-budget/ +title: Who should control the budget? +authors: +- tarikn +--- + + +

    Who is really ruling the country? Is it the political party with the most ministerial seats, or the one with the most influential ones? And how do we measure the relative weight of a ministry? + +

    Morocco’s government, as in many countries around the globe, consists of a coalition of political parties. Unlike the United States, no single party controls the government alone, which leaves the opportunity for negotiations and maneuvers to split the ministerial seats. For example, the Istiqlal Party recently began criticizing the draft of the 2014 budget law, despite having earlier authored it before deciding to leave the government and switch to the opposition. + +

    In the political lingo, democracy is to many people synonymous to ‘number of seats’. We are becoming familiar with terms such as ‘majority’ or ‘minority’ based on the number of ‘seats’ of the parliament. Last month, the Moroccan Head of Government, Mr. Benkirane, negotiated the formation of a new coalition government. His party, PJD, kept most of the ministerial seats since they were the majority party. Yet it was disturbing to analyze the budget data distributed by ministers’ political party affiliation: www.floussna.ma. Only 8% of the budget spending is directly controlled by the Ministers of the majority political party. More than half of the budget is under ministers with no political affiliation (e.g. Ministry of Interior, Ministry of Education, Ministry of Agriculture). + +

    One would wonder if the citizen should have any voice on who shall control the budget. Today, we are asking citizens to get engaged on democratic practices, vote for the best profiles and programs, and accept the rule of the majority. Though the majority party has most of the ministerial seats, they have no power over the economic agenda. Is this a matter of concern? Are there any international practices for tying the citizens’ vote with budget control? + +

    The budget discussion raises new questions about the meaning of representative democracy. These questions and others are warranting the distrust of politics in Morocco and the call for more transparency and accountability. + diff --git a/examples/openspending/content/blog/2013-11-09-northern-mariana-islands-retirement-fund-bankruptcy.md b/examples/openspending/content/blog/2013-11-09-northern-mariana-islands-retirement-fund-bankruptcy.md new file mode 100644 index 00000000..5f42a2d7 --- /dev/null +++ b/examples/openspending/content/blog/2013-11-09-northern-mariana-islands-retirement-fund-bankruptcy.md @@ -0,0 +1,28 @@ +--- +redirect_from: /2013/11/northern-mariana-islands-retirement-fund-bankruptcy/ +title: Northern Mariana Islands Retirement Fund Bankruptcy +authors: +- Rufus Pollock +--- + + +Back on April 17 2012 the Northern Mariana Islands Retirement Fund attempted to file for bankruptcy under Chapter 11. There was some pretty interesting reading in their [petition for bankruptcy][petition] including this section (para 10) which suggests some pretty bad public financial management (emphasis added): + +[petition]: http://www.nmiretirement.com/downloads/Docket_Files/08_First_Day_Declaration.pdf + +> "Debtor has had difficulty maintaining healthy funding levels due to a combination of factors, including: the **failure of the Commonwealth's central government and autonomous agencies to remit full employer contributions**; a difficult investing climate over the most recent three to four years; and **a benefit structure that has been continuously increased and made more generous by the Commonwealth government without a corresponding increase in funding to the Debtor to cover increased costs**." + +northern-mariana-island-retirement-declaration-pdf + +The Government then made things even worse: + +> "These issues have combined to place the fund in a dire financial situation. Compounding this perfect storm, the **Commonwealth Government has passed laws declaring payment holidays, diverting earmarked revenues from the Debtor and reducing contribution rates for the Commonwealth Government, its agencies and political subdivisions**." + +And they are now very bankrupt: + +> "As a result, as of the date of Debtor's last actuarial report as of October 1, 2009, Debtor's actuary determined that Debtor's obligation was approximately 38.8% funded. While the Debtor is currently engaged in compiling information necessary for an updated actuarial study, the Debtor estimates that it is currently 32% funded. Debtor's financial condition continues to deteriorate, and, by management's figures, the **Debtor will deplete its assets by July, 2014 and thereafter be unable to provide any level of benefits to current and future Beneficiaries**" + +Remember what it means when a pension fund goes bankrupt: it means all those people who worked, often for many years, will suddenly receive nothing, often in a period of their life when they most need support. + +Note that ultimately Marianas petition for bankdruptcy got refused - as a governmental unit they are not eligible for Chapter 11. More on this and its relation to Puerto Rico situation at + diff --git a/examples/openspending/content/blog/2013-11-14-new-features-in-october-2013.md b/examples/openspending/content/blog/2013-11-14-new-features-in-october-2013.md new file mode 100644 index 00000000..6df1fd89 --- /dev/null +++ b/examples/openspending/content/blog/2013-11-14-new-features-in-october-2013.md @@ -0,0 +1,62 @@ +--- +redirect_from: /2013/11/new-features-in-october-2013/ +title: New Features in October 2013 +authors: +- Tryggvi Björgvinsson +--- +Compared to September, October may seem a less active month in terms of development. However, though there were fewer features in October they were bigger and took longer to develop. Let's just dive in. + +![Image by CarbonNYC (cc-by 2.0)](http://farm4.staticflickr.com/3245/2294144289_a54db90ac5_z.jpg "Security without obscurity") + +### Increased security + +One of the most important features we added in October was SSL certificates, or access via HTTPS. This means we can now encrypt communications between you and the [openspending.org](http://openspending.org). + +When you use websites you usually send things like cookies and headers in plain, text (visible for those who want to see it). Cookies are for example used to store who you are and let the server know. If these are sent in plain text, other people can pretend that they are you. Remember how [Firesheep](http://codebutler.com/firesheep/) was used to gain access to people's accounts on Facebook, GMail, etc.? + +When using our API users can authenticate via the headers they send. When people from CERN added the Data Loading API to OpenSpending back in September they wanted it to be more secure, so instead of just sending your API key you had to use a private api key to sign your request. If we hadn't done that, other people would have been able to just grab your API key from the plain text header and then start uploading data as you which could cause you some problems. + +With HTTPS all that gets encrypted so it'll be harder for those bad people to pretend they are you (and probably not worth the effort since they'd have to break the same security measures as online banking uses). + +Now this is not perfect yet. We don't enforce the use of HTTPS yet so you will have to ask [openspending.org](http://openspending.org) to serve you the content over HTTPS. We are preparing to enforce traffic over HTTPS in November so you can be secure without even thinking about it (you actually should never stop thinking about security, but this will make your life easier). Until then you can use browser plugins like [HTTPS Everywhere](https://www.eff.org/https-everywhere) that try to use HTTPS first. + +When we have the enforced HTTPS we will fall back to the old version of the API authentication because the communications will be secure enough to drop your API key into the header without compromising your account. Easier and more secure times ahead. + +### Faster dataset index + +One of the features we introduced [in July and August](http://community.openspending.org/2013/09/features-in-july-and-august-2013/) was a dataset page. The reason for it was that it took a long time to load all of the datasets in OpenSpending so when users clicked on the Datasets link in the navigational bar, nothing happened (even though behind the scenes the browser was just waiting for a response). + +We decided to go for a quick fix back in July and August and redirect people to a specific dataset page instead of doing things behind the scenes. That way users would at least see that the browser was working. We also acknowledged that this was not the best solution and invited you to help us out. + +We're now proud to say that the dataset page has been improved. We're still using the dedicated dataset page *but* we have put it behind a cache so it's way faster. We went from the 15 seconds (and up to 24 seconds) measured down to around 0.3 seconds! That's an awesome improvement and if you've ever doubted caching, this is one good reason to use it. + +Now there are still tweaks that we could do. For example when the dataset list updates (e.g. when somebody publishes a new dataset) we need to regenerate the cache. We don't do that automatically at the moment so the first user to visit the dataset page will have to wait all those seconds for anything to happen. We would like to autogenerate the new cache so you'll never have to wait for seconds. + +There are two approaches we could take. One would be to find the bottleneck in the dataset index functionality itself, the other would be to regenerate on dataset publication and serve the older version until the new one is ready. + +**Would you like to help us out?** You can offer your help as both a code reviewer or a coder. We can help you get started. Just let us know you’re interested on our [developer mailing list](http://lists.okfn.org/mailman/listinfo/openspending-dev) or our IRC channel, *#openspending on irc.freenode.net*! + +### Bosnian budget analysis + +For quite some time we have been working on a satellite site, similar to [Where does my money go?](http://wheredoesmymoneygo.org) for Bosnia and Herzegovina, [Budzeti.ba](http://budzeti.ba). In October we finally launched the Bosnian site! If you look at these pages (Where does my money go? and Budzeti.ba) you'll see similar functionality but there are a lot of special things about Budzeti.ba. + +The country is divided into three entities (they actually have rotating presidents from these three entities) and one of these entities is further divided into cantons. The tax system is different in each of the three entities and people look at salary and taxes differently there than in the United Kingdom. + +So we had to make some big changes to the daily bread visualisation, which in their case became a tax calculator (not showing the daily taxes, but just how the taxes are split). We took the chance and redid how we created the colours in the visualisation so the bubbles now follow the same colour scheme as in the bubbletree (on *Where does my money go?* all of the bubbles were purple). + +We also commented the code and hopefully made things more simple for those who want to put up a daily bread visualisation on their site. If you're interested in the code behind the page, it's hosted as a github page and the code is accessible from [that repository](http://github.com/openspending/budzeti.ba). + +**Are you setting up your own satellite site?** Join the [developer mailing list](http://lists.okfn.org/mailman/listinfo/openspending-dev) or our IRC channel, *#openspending on irc.freenode.net* and we can help you out if you run into some problems. + +### Other Changes + +As always there were smaller changes made to OpenSpending. These other changes were mostly related to the three big changes mentioned above. Things like restructuring the code around the caching mechanism, moving assets behind a CDN (content delivery network), new taxman jurisdictions (actually added a while back) and preparations for the new OpenSpendingJS visualisation library. + +Remember that you can help develop OpenSpending on many fronts, be it the core OpenSpending platform, OpenSpendingJS, Taxman, the satellite site template or whatever. Just have a look at our [OpenSpending issue tracker](http://github.com/openspending/openspending/issues) or our [OpenSpendingJS issue tracker](http://github.com/openspending/openspendingjs/issues), that's where most of the issues are being tracked. Feel free though to add features you think are missing in any project and become a contributor to the OpenSpending project. + +### Thanks + +Thanks to **Jiri Kuncar**, **Alberto Rodriguez Peon**, **Joel Rebello**, and **Nick Stenning** for their contributions (there are probably a lot more who've contributed somehow to OpenSpending so don't be sad if we forgot you - it's a bit hard to manage, just let us know and we'll add you). + +Image of padlock used in this blog post is by [CarbonNYC](http://www.flickr.com/photos/carbonnyc/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). + diff --git a/examples/openspending/content/blog/2013-11-21-launching-spending-stories-how-much-is-it-really.md b/examples/openspending/content/blog/2013-11-21-launching-spending-stories-how-much-is-it-really.md new file mode 100644 index 00000000..a62c71b3 --- /dev/null +++ b/examples/openspending/content/blog/2013-11-21-launching-spending-stories-how-much-is-it-really.md @@ -0,0 +1,36 @@ +--- +redirect_from: /2013/11/launching-spending-stories-how-much-is-it-really/ +title: 'Launching Spending Stories: How much is it really?' +authors: +- Neil Ashton +--- +spendingstories + +Spending Stories is a new way to put spending figures in their proper perspective. Developed by the Open Knowledge Foundation and Journalism++ with funding from the Knight Foundation, Spending Stories is an app that helps citizens and journalists understand and compare amounts in stories from the news. + +When we hear that the UK's school meals programme costs £6 million, what does that really mean? It means, for one thing, that it costs about a fifth of the annual spending on the monarchy. + +Spending Stories draws out comparisons between amounts of money, giving users a context in which to understand how money is being spent across society while referencing the original news stories. + +Users can enter a figure into Spending Stories and get a scale visualisation showing how it compares with spending stories from the app's database. + +£700,000: scale visualisation + +The app displays the big picture, and users can then click through to a card visualisation that shows how the amount relates to specific stories. + +£700,000: card visualisation + +Users can filter stories to only show amounts that relate to the user's interests, for example aid or energy. + +Filtering stories + +If users find news stories of interest, they can contribute these to the database in three easy steps and share them. + +Contribute new data + +Due to the good availability of UK spending data in OpenSpending, this first release of Spending Stories focuses on the UK. Spending Stories is, however, an open source project and can easily be forked and translated into other languages. + +We hope to help Spending Stories sites launch on their own and expand with new features and local news stories. At launch, we are already in touch with Open Knowledge Foundation Japan about the potential deployment of Spending Stories in Japanese. + +If you would like to know more about the options for setting up a local Spending Stories site, get in touch. + diff --git a/examples/openspending/content/blog/2013-11-25-linkedspending-openspending-becomes-linked-open-data.md b/examples/openspending/content/blog/2013-11-25-linkedspending-openspending-becomes-linked-open-data.md new file mode 100644 index 00000000..885db27d --- /dev/null +++ b/examples/openspending/content/blog/2013-11-25-linkedspending-openspending-becomes-linked-open-data.md @@ -0,0 +1,45 @@ +--- +redirect_from: /2013/11/linkedspending-openspending-becomes-linked-open-data/ +title: 'LinkedSpending: OpenSpending becomes Linked Open Data' +authors: +- Anders Pedersen +--- +![Image by the Linkedspending Project](http://linkedspending.aksw.org/extensions/page/page/img/rdfdatacube_shrinked.png) + +The LinkedSpending project converts the OpenSpending data to RDF and publishes the resulting data. This data is represented in the RDF Data Cube format and is freely available and openly licensed. + +## Conversion +All of the OpenSpending datasets describe observations referring to a specific point or period in time and thus undergo only minor changes. +New datasets however, are frequently added. Because of this, the huge number of datasets and their size, an automatic, repeatable transformation is required. +This is realized by an [open source Java application](https://github.com/AKSW/openspending2rdf) which fetches a list of datasets on execution and only transforms the ones who are not transformed yet. Datasets are downloaded as JSON over the OpenSpending API and are cached to prevent unnecessary traffic. + +For each class used by LinkedSpending, there is a mapping at which URL the information used to create the instances of those classes is found. Because the source data adheres to the data cube model, a conversion of the data to RDF needs an appropriate RDF vocabulary. The RDF DataCube vocabulary, i.e.an RDF variant of the data cube model, is an ideal fit for the transformed data. + +## Publishing + +LinkedSpending data is published using [an instance](http://linkedspending.aksw.org) of the [OntoWiki](http://aksw.org/Projects/OntoWiki.html) which is a Wiki for Semantic Information.Depending on the actor and the needs, OntoWiki provides various abilities to gather the published RDF data as described as follows. +The data can be explored by viewing the properties of a resource, its values and by following links to other resources. Using the [SPARQL endpoint](http://linkedspending.aksw.org/sparql), actors are able to satisfy complex information needs {% raw %}(example query).{% endraw %} +Faceted search offers a selection of values for certain properties and thus slice and dice of the dataset according to the interests on the fly. + +## Usage Scenarios +Apart from the information needs that can already be satisfied using the source data, there are usage scenarios that are easier or only possible with LinkedSpending. + +### Economic Analysis +LinkedSpending is represented in Linked Open Data which facilitates data integration. +Currencies and countries from DBpedia and LinkedGeoData, respectively, are already integrated. +This allows to create SPARQL queries for information needs such as "which datasets use currencies whose inflation rates are greater than 10 %". + +### Finding and Comparing Relevant Datasets +Government spending amounts are often much higher than the sums ordinary people are used to dealing with but even for policy makers it is hard to understand, if a certain amount of money spent is too high or normal. +Comparing datasets and finding those which are similar to another one helps separating common values from outliers which should be further investigated. For example, if another country has a similar budget structure but spends way less on healthcare with a similar health level. + +## Future Work +The improvement of LinkedSpending will be the task of a software engineering internship group which is part of the bachelor degree program at our University. OntoWiki is developed in research group. Some of the plans are: + +- analyze the data quality +- integrate LinkedSpending with further datasets +- improve faceted search and visualization + +## Contact + +LinkedSpending is a project of the research group "Agile Knowledge Engineering and Semantic Web" (AKSW) which is hosted by the Chair of Business Information Systems (BIS) of the Institute of Computer Science (IfI) / University of Leipzig as well as the Institute for Applied Informatics (InfAI). Please see the [paper draft](http://svn.aksw.org/papers/2013/openspending2rdf/public.pdf) for further details or [contact us](mailto:hoeffner@informatik.uni-leipzig.de). diff --git a/examples/openspending/content/blog/2013-11-27-budgit-an-app-and-platform-to-track-service-delivery.md b/examples/openspending/content/blog/2013-11-27-budgit-an-app-and-platform-to-track-service-delivery.md new file mode 100644 index 00000000..b71127f3 --- /dev/null +++ b/examples/openspending/content/blog/2013-11-27-budgit-an-app-and-platform-to-track-service-delivery.md @@ -0,0 +1,24 @@ +--- +redirect_from: /2013/11/budgit-an-app-and-platform-to-track-service-delivery/ +title: 'BudgIT: An App and Platform to track service delivery' +authors: +- Oluseun Onigbinde +--- +

    I swim in an innovation pool that boasts of developers, interface designers, programmers and entrepreneurs called Co-Creation Hub. Located in the emerging tech district in Lagos, Co-Creation Hub is  the geek’s nest and the innovator’s hub. I am a faithful of that academy that picked me up from the bank’s cubicle to lead BudgIT, the double helix of civic awesomeness. + +

    BudgIT strives hard to make budget access easy for everyone. Few months ago, we had our website with a messy backend and our mobile platforms were grossly underperforming.  The only bright spot was our Twitter page which was a restless stream of tweets on public finance and other related data. Working with a group of amazing geeks in Co-Creation Hub, BudgIT has been able to fix it platforms with every state having its personalized page which include infographics, interactive applications and relevant data. + +

      + +

    Our mobile platform experience is different from the web and we have gone a mile further to release Android and Blackberry Apps. We have gone ahead to redefine the narrative on our mobile platforms with  quick access to the budget, projects and monthly allocations. We have also released our apps on the Blackberry and Android platforms,  which are the most common within our environment.  This is a way of reaching to young and urban Nigerians who have access to smartphones and can lead the national discussion on budgets and public data. + +

    Screenshot of BudgIT Android App + +

    Screenshot of BudgIT mobile site (yourbudgit.com) + +We have decided to go from a budget access to budget tracking platform. This new offering of ours is known as Tracka. Tracka is a social platform of active citizens who are interested in tracking budgets and public projects in their community. Layered on open data and also integrated with existing social media tools, this platform will bring people of common interests together to share photos, videos, documents and also post comments on existing projects. This has the power extend the use of open data to the larger society who earnestly yearn for improved government services. There is a strong wish from our end to integrate Tracka with OpenSpending, the visualization platform of the Open Knowledge Foundation. + +Tracka: Our Public Projects Platform + +Our goal is to amplify the voices by shining light in corners less understood. Most people are not aware of projects in the neighbourhood and don’t ask the right questions or connect to institutions you are suppose to implement the projects. BudgIT wants to lead that conversation based on facts that the budget is a promise to the citizens. Performance can only be ensured when the citizens and government are on the same page. We want to match the right communication tool to every citizen and ensure that they become active citizens by actively demanding what is right for them. The next push will be spike the kind of viral marketing, the kind the Nigerian ecommerce platforms put forward in their early days. As we go on the path of democracy, every voice must count. + diff --git a/examples/openspending/content/blog/2013-12-01-help-us-map-the-spending-projects.md b/examples/openspending/content/blog/2013-12-01-help-us-map-the-spending-projects.md new file mode 100644 index 00000000..da8e7c47 --- /dev/null +++ b/examples/openspending/content/blog/2013-12-01-help-us-map-the-spending-projects.md @@ -0,0 +1,19 @@ +--- +redirect_from: /2013/12/help-us-map-the-spending-projects/ +title: Help map spending projects around world! +authors: +- Anders Pedersen +--- +OpenSpending data is used in projects across the globe by students, journalists, activists, news media, governments, and others. Browse the map to discover some of these exciting projects, and get ideas for your own spending tracking projects. + +News Editor Anna Flagg has now created an upgraded map of spending projects from around the world. The map includes investigative projects, budget monitoring groups and financial transparency initiatives. We need your help to expand the map with some of all the new and fantastic projects out there. Here is a quick guide for how you can add one or more projects in less than five minutes: + +1) Go to the google doc of spending projects, which functions as the "backend" of the map. Ask for approval to become editor in the right corner [the blue coloured "share" button] + +2) Add a financial project with information about the country, the URL for the project and other details. If you are unsure about how to add the information, get in touch via the OpenSpending mailing list + +3) Share your contribution to the map by tweeting and emailing the mailing list! + +*Made by Anna Flagg using D3.js.* + +[iframe src="http://www.annaflagg.com/os" frameborder="0" width="620" height="320"] diff --git a/examples/openspending/content/blog/2013-12-06-processing-data-for-open-spending.md b/examples/openspending/content/blog/2013-12-06-processing-data-for-open-spending.md new file mode 100644 index 00000000..38627a2e --- /dev/null +++ b/examples/openspending/content/blog/2013-12-06-processing-data-for-open-spending.md @@ -0,0 +1,40 @@ +--- +redirect_from: /2013/12/processing-data-for-open-spending/ +title: Processing data for Open Spending +authors: +- Olaf Veerman +--- +*[Flipside](http://flipside.org/) is a Lisbon-based web development agency that recently added over 380,000 non-bid contracts from the Portuguese government to OpenSpending. This guest post by [Daniel Silva](https://twitter.com/danielfdsilva) describes how it was done.* + +Even though the data on non-bid expenditures from the Portuguese government is already publicly available on [Base.gov.pt](http://www.base.gov.pt), data on OpenSpending is much more accessible. People can explore and browse the set in the browser or use the API to create visualizations or other insights. You can find the [Portuguese contract data on OpenSpending here][os-data]. + +[os-data]: https://openspending.org/pt_ajustes-diretos + +## Processing the data + +To prepare the data for import, we used Python to process its files, with one JSON document per line, and turned the data into a CSV. The most important things the script handles are: + +- all dates are formatted to `yyyy-mm-dd` +- whenever the `signing date` is empty, the field is populated with the `publication date` (Openspending discards any row with empty cells) +- the fields containing amounts are stripped of the Euro sign and the thousand-separator, and the decimal mark is set to `.` +- the CPV code is split from its description +- multiple locations are split with a pipe +- multiple contracting or contracted entities are combined in one new entity. + +You can check the code for yourself on [the processor's Github repository](https://github.com/flipside-org/ajustes-processor). + +### Multiple contracting or contracted entities + +OpenSpending doesn't support multiple contracting or contracted entities for one record. The format used for importing is CSV, and since it has a flat structure, it doesn't allow multidimensional values. + +To solve this problem, we have two options: + +- Split the contracting and contracted entities, creating a contract for each one and dividing the total amount by the number of entities. This would lead to several duplicate contracts and erroneous amounts, since not every contracted company will be paid the same. +- Keep one single contract and merge the entities. The problem with this approach is that analysis of relationships between entities becomes more difficult. + +We chose the latter approach so the contract maintains its integrity. Users that want to further analyse the dataset will always be able to preprocess the data and split entities before doing so. + +## Creating the model + +Besides preparing the CSV file for import, OpenSpending also requires people to create a model for the dataset. We decided to add as much information as possible, even though we were forced to leave out some potentially interesting data regarding amounts actually spent. Since not all contracts contain these amounts and OpenSpending discards rows with empty dates or floats, we decided to not include them in the model just yet. You can [check the model](https://openspending.org/pt_ajustes-diretos/model.json) on openspending.org. + diff --git a/examples/openspending/content/blog/2013-12-11-big-lottery-fund-grants-data-on-openspending.md b/examples/openspending/content/blog/2013-12-11-big-lottery-fund-grants-data-on-openspending.md new file mode 100644 index 00000000..f759d65b --- /dev/null +++ b/examples/openspending/content/blog/2013-12-11-big-lottery-fund-grants-data-on-openspending.md @@ -0,0 +1,28 @@ +--- +redirect_from: /2013/12/big-lottery-fund-grants-data-on-openspending/ +title: Big Lottery Fund Grants Data on OpenSpending +authors: +- Rufus Pollock +--- +The Big Lottery Fund have been doing some great strides with [open data](http://okfn.org/opendata/) publishing all of their historical grants data as open data at: + +This seemed [perfect for OpenSpending](https://github.com/openspending/datatoload/issues/29) and so with a few minutes of data wrangling we've got the data into OpenSpending (only 2012/2013 data so far): + + + +You can already get some nice overviews, for example + +### By local authority then grantee + +Note: this is just 2013 data (not 2012 + 2013) + +big-lottery-fund-grants-local-authority-grantee-os + +### By region then local authority then grantee + + + +### Create your own + +You can create your own at: + diff --git a/examples/openspending/content/blog/2013-12-12-new-features-in-november-2013.md b/examples/openspending/content/blog/2013-12-12-new-features-in-november-2013.md new file mode 100644 index 00000000..2c32aa9c --- /dev/null +++ b/examples/openspending/content/blog/2013-12-12-new-features-in-november-2013.md @@ -0,0 +1,112 @@ +--- +redirect_from: /2013/12/new-features-in-november-2013/ +title: New Features in November 2013 +authors: +- Tryggvi Björgvinsson +--- +It's so exciting to participate in the OpenSpending project. We continue to raise the bar when it comes to what we as a community can do and that's just what we did in November. The community is growing and achieving more (as you do when you grow) and this is what makes OpenSpending so exciting to be a part of right now. + +![Image by epSos.de (cc-by 2.0)](http://farm9.staticflickr.com/8515/8474532085_6d010ee8d0_z.jpg "OpenSpending is global and local just like money") + +### Translations + +Sadly we haven't been putting much focus on translations in the last months but that changed in November. We put a lot of effort into internationalising (often abbreviated as *i18n*), which means we put effort into making OpenSpending better at handling different global communities. The biggest effort would be marking strings as ready for translation, which we did. This had been done before but a lot of the strings weren't marked so we ended up with 150% more strings that needed translation. + +The second step in the process is the localisation (often abbreviated as *l10n*), which means doing the adaptation to a specific region or culture. Again the biggest effort there are translations. Going through the strings we marked in the previous step and translating them into a specific target language (locale). The awesome thing is that, that's exactly where our community shines! + +Japanese was the first complete translation (100% of all strings translated), not counting English. The most impressive thing was that the Japanese translation team finished the translations only five days after we pushed out the new strings! Bosnian was also quite impressive. It went from zero to more than half of the strings translated in a single day thanks to a single translator! Italian has been steadily getting more and more translations and now has the second most translated strings thanks to a translator who also does French. These all happened in November, just days after we pushed the new strings so it's going to be really fun to watch how we grow in translations in the coming months. + +Translations are really important for a global project like OpenSpending. **If you want to help us reach as many people as possible** you can help us translate OpenSpending into your language. Just go to our [project page on Transifex](https://www.transifex.com/projects/p/openspending/) and offer help with translations into your language (or ask us to create your language if it isn't there already). When accepted, just jump in and start translating OpenSpending. + +### Jinja2 migrations + +One of the reasons we were able to push a lot of the new strings for translations was that we finished the migration to Jinja2. Back in September we wrote about how changes in July and August had caused OpenSpending to have two navigational bars. That was because we were in the middle of a transition from the [Genshi](http://genshi.edgewall.org/) template system to [Jinja2](http://jinja.pocoo.org/). Pages served with Genshi had a different navigational bar than those served with Jinja2. + +We have now completed the migration so we're only serving a single version of the navigational bar on OpenSpending. Besides being less confusing for the user and improving *i18n* the Jinja2 templates are simpler for most developer which means we're now even more newcomer friendly than before (there's always room for improvement). We hope that this will drive more people to help us out developing and maintaining the the HTML output of OpenSpending. + +**Are you good at or interested in creating web pages?** You can help us make OpenSpending in the browser work better for the users and look better. We use Jinja2 so it's easy as eating a pie to get started and help out. Take a look at our [howto hack](http://community.openspending.org/help/development/volunteer/) for pointers on how to dive in (or offer help). + +### Measures and metrics + +It's not enough to just have an easy to use template system and an engaged community. We also need to know what to change. That's why we decided to start watching you... no, not in the *NSA revelations* way. We just created some measurements to track how users use OpenSpending. What are they looking for and how do they get there? This can then feed into a site redesign to serve users of OpenSpending better, instead of just randomly changing things and hoping it's good. + +We do, of course, have this sneaking suspicion that users go to OpenSpending to either find or upload spending datasets. That's why we also did slightly more than just track mouse clicks and site visits. We created a user scoreboard which is only available for administrators on OpenSpending. The scoreboard is pretty simple. What it does is see how many datasets users are connected to. This helps us analyse things like, are all dataset uploaded by an elite group of users (who might have some bias) or is there a good diversity of users uploading the datasets (the latter being more beneficial to OpenSpending). The scoreboard also shows us how many users are registered but still haven't uploaded any datasets. This is an indicator that people might find uploading datasets difficult (which we can then tackle). + +So basically we're just analysing how people use the site to identify problems in order to address them and improve the experience for everyone. + +### Community home + +We already made our first change and we actually didn't need any analysis to do it because it was really straightforward and obvious that it had to be done. + +Back in September, we finalised our community page when we renamed it from blog.openspending.org to community.openspending.org because that's what it is, the home of the OpenSpending community. + +The problem was that community.openspending.org hadn't really been integrated into the OpenSpending platform at [openspending.org](https://openspending.org) so in November we finally did just that. We updated the links in the navigational bar to go to different community pages instead of serving content via [openspending.org](https://openspending.org) and we removed all content except the dataset management aspect and the front page from [openspending.org](https://openspending.org) and just redirect people to community.openspending.org instead. + +So we're creating a better separation between the platform and the community. The community is much bigger than a single piece of software. We also updated the community.openspending.org site so that instead of serving the blog on the front page we now have a landing page that helps people find what they're looking for without having to guess where in the navigational bar on the community site it is. + +The control of the community home is now more in the hands of the community itself instead of some developers. So now developers can now also focus more on improving the tool itself, instead of guiding community members through a technical maze of how to contribute documentation, blog posts or whatever the community as a whole wants to do in their cyberhome. + +### Development community growth + +The OpenSpending community is growing fast. One concern we have is that the developer sub-community, which hasn't been growing as fast (but still growing) won't be able to handle this growth efficiently which would be a big shame. So we are now going to try and reach out to prospective developers who might be interested in our mission and let them know about us. + +In November we took a small step in that direction. We put up a profile for OpenSpending on [OpenHatch](http://openhatch.org/projects/openspending) which is a project that is *"dedicated to matching prospective free and open source contributors to communities, tools, and education."* There are a lot of interesting projects on OpenHatch so we have some competition on the site, but since OpenSpending has a mission to make the world better for everyone, we have a good chance of getting people interested. We are making efforts of being newcomer friendly. We just need some visibility. + +**You can help us reach out to developers**. Do you know people who might be interested in developing OpenSpending? Let them know about us and help us get them engaged! You can direct them to our community or you can point them to [our OpenHatch page](http://openhatch.org/projects/openspending). + +If you know about any other outreach programs where we can find prospective developers, let us know or help us get in there. We are newcomer friendly and we are an exciting project. This way we all put in a helping hand to help our community grow by helping our development community grow. + +### Permissions API + +On more technical notes, we created a new API resource to check for permissions a user has on datasets in OpenSpending. Checking for permissions might sound a little bit weird but this enables external tools to check if their users (who are also users on OpenSpending) have permission to create, read, update or delete a dataset *before* they do something. This can, for example, save a lot of bandwidth because the external tools can avoid downloading big datasets in vain because a user didn't have the right permissions. + +We are already working on a project, called [os-upload](https://github.com/openspending/os-upload), which allows users to upload datasets on a different page which then sends the datasets onward to OpenSpending. + +Currently this uses the API key of users to interact with the site on the users behalf (so that the datasets will be uploaded as the user, not the site). The API key is used to check for permissions and then it's used again to upload the datasets. + +Note that you should *avoid* sharing your API key with anybody since the API key allows others to pretend that they are you. We will run os-upload as part of OpenSpending, wrapping it under an SSL certificate for secure communications. We're working on this now but until then we recommend you to test things out with a test account rather than your real account. Going via HTTPS (SSL certificates) is of course not the ultimate security we want but at least we'll mitigate the risk that somebody gets your API key and it's just good to have around. + +**We need your help!**. If you know your way around authentication for external sites without giving out your credentials or API key (for example via [OAuth](http://oauth.net/) or another way) get in touch and help us make OpenSpending and external sites more secure. + +### Enforced Security + +Talking about security and passing API keys around, we decided to make HTTPS the default *and* only way of interacting with [openspending.org](https://openspending.org), just like we plan to do with with os-upload. In October we increased our security by adding SSL certificates and HTTPS to [openspending.org](https://openspending.org). In November we enforced it. If you go to **http**://openspending.org you will be automatically redirected to **https**://openspending.org and a secure communication channel established. + +This way we can, with higher confidence, use API keys for authorisation and be less afraid that someone might be able to pretend to be someone else on OpenSpending. It's just all around better to have security by default instead of having it optional. + +### Simplified API authentication + +The enforced security allowed us to simplify the API authentication which was developed for the data loading API. If you wanted to load data via the API you had to use two API keys, one public and one secret. You had to sign your request with your private key which OpenSpending was then able to verify to be sure you were you. + +With the enforced security you now only need one API key which you put into the request header (without doing any computations or processing) and OpenSpending can use that API key to look you up. Of course this is why it's important that you don't share your API key with anybody because that means they'll be able to pretend that they're you. + +But now it's way simpler to load data into OpenSpending. Bring on the datasets! + +### Taxman standardisation + +We touched a lot of important aspects of software development that don't involve coding in November. Translations was a big thing but we also did documentation. The biggest effort in documentation was the first attempt to standardised how Taxman should work. + +We created a page on the [Taxman wiki](https://github.com/openspending/taxman/wiki/Api-conventions) to discuss and try to find the common API for all jurisdictions. This is a first stab and trying to create this common API based on the jurisdictions we already have. We expect this to evolve over the coming months and in the end we'll have a simple, generic way to access all of the jurisdictions in Taxman. + +As it says in the [Taxman README](https://github.com/openspending/taxman/blob/master/README.md): + +> "At the moment we place no restrictions on what you return from the +> calculate function. That said, it is hoped that as we add more jurisdictions +> we will work out which parts of the API we can standardise. Consistency +> across jurisdictions is very important if TaxMan is to be useful, and at the +> moment we are relying entirely on contributors' discipline to ensure it." + +We're now heading into the standardisation phase which is very exciting. **Do you want to help?** We need input about all jurisdictions so we can get this right. You can read through this to see if we're missing something important for your jurisdiction (or you can even implement the calculations for your jurisdiction according to the common API to see if it works). Just hop on to the [Taxman repo](https://github.com/openspending/taxman/) and dig in. + +### Other Changes + +Of course there are other changes we did in November. We, for example, updated READMEs to make the information a bit clearer for users and developers, created some documentation (on community.openspending.org) and worked a lot on the [os-upload](https://github.com/openspending/os-upload) project. + +Remember that you can help develop OpenSpending on many fronts, be it the core [OpenSpending platform](https://github.com/openspending/openspending), [OpenSpendingJS](https://github.com/openspending/openspendingjs), [Taxman](https://github.com/openspending/taxman), [the satellite site template](https://github.com/openspending/satellite-template) or any other software project in the [OpenSpending project list](https://github.com/openspending/). Just have a look at our [OpenSpending issue tracker](https://github.com/openspending/openspending/issues) or our [OpenSpendingJS issue tracker](https://github.com/openspending/openspendingjs/issues), that's where most of the issues are being tracked. Feel free though to add features you think are missing in any project and become a contributor to the OpenSpending project. + +### Thanks + +**Everton Zanella Alvarenga**, **Nigel Babu**, **Maya Barisic**, **Lucy Chambers**, **Tajima Itsuro**, **Yoshihide Jimbo**, **kawando**, **Martin Keegan**, **Iwao Kobayashi**, **Elisabetta Lombardo**, **masanori.katsuragawa**, **Takano Mitsuhiro**, **Prakash Neupane**, **Takashi Nishibayashi**, **nyampire**, **高木祐介**, **Anders Pedersen**, **Rufus Pollock**, **Joel Rebello**, **rigg**, **Toshihide Sato**, **Hal Seki**, **Nick Stenning**, **tosseot**, and **William Waites**. This was quite a big month in terms of development help so there are probably a lot more who have contributed somehow to OpenSpending. Just let us know and we'll add you to the list. + +Image of different currencies used in this blog post is by [epSos.de](http://www.flickr.com/photos/epsos/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). + diff --git a/examples/openspending/content/blog/2014-01-01-the-spending-data-handbook-revised-and-updated.md b/examples/openspending/content/blog/2014-01-01-the-spending-data-handbook-revised-and-updated.md new file mode 100644 index 00000000..bef35b08 --- /dev/null +++ b/examples/openspending/content/blog/2014-01-01-the-spending-data-handbook-revised-and-updated.md @@ -0,0 +1,21 @@ +--- +redirect_from: /2014/01/the-spending-data-handbook-revised-and-updated/ +title: The Spending Data Handbook, revised and updated +authors: +- Neil Ashton +--- +[caption width="500" align="aligncenter"] Photo credit: Anthony Joh[/caption] + +The [Spending Data Handbook](http://community.openspending.org/research/handbook) is a user's manual for government budget and spending data. If you want to learn what spending data is and how to work with it—whether you're a journalist, a member of a civil society organization, or just an interested citizen—look no further than the Spending Data Handbook. + +In the Handbook, you'll find answers to questions like the following: + +* **Data literacy**. What do we mean by "data" and "spending data"? What's the big deal with "machine-readable data"? +* **Working with data**. What is the data-wrangling workflow? How do you find and clean up a dataset? +* **Presentation and engagement**. How do you engage an audience with your data's message? What are some nice examples of spending data projects? +* **Working with others**. Why take a collaborative approach to data-driven research? How can you make the case for open data to governments? + +The first version of the Handbook was produced in a four-day [writing sprint](http://booksprints.net/) that brought open data hackers and civil society advocates together to share their perspectives on government spending data. The Open Knowledge Foundation has now edited and reorganized the results of that sprint to tighten the Handbook's overall flow and presentation. + +The Spending Data Handbook is released under a Creative Commons attribution licence. Anyone is free to use and reuse its contents. But more than that, readers are free to translate, revise, and expand the handbook as they see fit. The [Markdown manuscript of the Handbook](https://github.com/openspending/spending-data-handbook) is available on GitHub—just fork the repo to get started on revising or updating the Handbook. + diff --git a/examples/openspending/content/blog/2014-01-03-mapping-the-open-spending-data-community.md b/examples/openspending/content/blog/2014-01-03-mapping-the-open-spending-data-community.md new file mode 100644 index 00000000..f7c1f9e7 --- /dev/null +++ b/examples/openspending/content/blog/2014-01-03-mapping-the-open-spending-data-community.md @@ -0,0 +1,28 @@ +--- +redirect_from: /2014/01/mapping-the-open-spending-data-community/ +title: Mapping the Open Spending Data Community +authors: +- Neil Ashton +--- +[caption id="attachment_1779" align="aligncenter" width="669"]Mapping the Open Spending Data Community Mapping the Open Spending Data Community[/caption] + +We're pleased to announce the official release of "[Mapping the Open Spending Data Community](http://community.openspending.org/research/mappingcommunity/)" by [Anders Pedersen](http://okfn.org/members/anderspeders/) and [Lucy Chambers](http://okfn.org/members/lucychambers), an in-depth look at how citizens, journalists, and civil society organisations around the world are using data on government finances to further their civic missions. + +The investigation began in 2012 with three goals: + +* To identify Civil Society organisations (CSOs) around the world who are interested in working with government financial data +* To connect these CSOs with each other, with open data communities, and with other key stakeholders to exchange knowledge, experiences, and best practices in relation to spending data +* To discover how CSOs currently work with spending data, how they would like to use it, and what they would like to achieve + +This report is the result. It brings together key case studies from organisations who have done pioneering work in using technology to work with public finance data in each of [budgets][budgets], [spending][spending], and [procurements][procurements], and it presents a curated [selection of tools][tools] and other advice in an [appendix][appendix]. + +[budgets]: http://community.openspending.org/research/mappingcommunity/case-studies-budgets/ +[spending]: http://community.openspending.org/research/mappingcommunity/case-studies-spending/ +[procurements]: http://community.openspending.org/research/mappingcommunity/case-studies-procurements/ +[tools]: http://community.openspending.org/research/mappingcommunity/appendix/tool-ecosystem/ +[appendix]: http://community.openspending.org/research/mappingcommunity/appendix/ + +As part of this research, we've also produced a four-part video series "[Athens to Berlin][a2b]", which you can watch to meet some of the fascinating characters in the world of CSOs working with government spending data and to learn firsthand about their successes and their challenges. + +[a2b]: http://community.openspending.org/research/mappingcommunity/introduction/videos/ + diff --git a/examples/openspending/content/blog/2014-01-07-new-features-in-december-2013.md b/examples/openspending/content/blog/2014-01-07-new-features-in-december-2013.md new file mode 100644 index 00000000..4ea11a55 --- /dev/null +++ b/examples/openspending/content/blog/2014-01-07-new-features-in-december-2013.md @@ -0,0 +1,100 @@ +--- +redirect_from: /2014/01/new-features-in-december-2013/ +title: New Features in December 2013 +authors: +- Tryggvi Björgvinsson +--- +Happy New Year OpenSpending! The last month of 2013 was exciting. Even though the community took some time off around the holidays we still managed to squeeze in some really great developments and some big releases. + +![Image by mattcornock (cc-by 2.0)](http://farm8.staticflickr.com/7440/8801652469_0ca89fa40c_z.jpg "We have bar charts!") + +### Commandline loading + +Our loading API got some attention this month when one community member wanted to load data into OpenSpending directly from the commandline. That caused some problems so we needed to do some modifications, both to our documentation which wasn't clear enough and also some code changes. + +After those changes we were finally ready for commandline loading with the help of the wonderful [cURL](http://curl.haxx.se/) software. How you may ask. It's as easy as doing: + + curl -X POST -H "Authorization: ApiKey " --data "metadata=" + +If you have a lot of similar data to upload you can now script it with a simple commandline call like that (or a similar one from your favourite scripting language). + +Just like the community member said when we finished: + +> "Time to load some data..." + +### Translations + +We continue to provide better international support to our users. In December, two new languages went up to 100%: Nepali and Russian. The teams behind those translations did a great job. The Nepali translation went from zero to complete translation in only a few days while the Russian had slightly bigger problems. For example, they didn't have a Russian word for *dataset* (which is a pretty important term in OpenSpending). The Russians are even taking a step further and reviewing the translations to ensure the quality of the translations. This is a precedent other translators can take up, even though we won't enforce it. + +Other climbers on the translation list in December were Czech, Italian (both up to over 60%), Indonesian which is not far from surpassing German (which had been completely translated before we uploaded the new strings back in November) and French, which is not far away from Indonesian. + +So, we can start to get excited about some more complete translations soon. Will it be one of the more recent translation teams, one of the older teams, or a completely new language -- never before seen in OpenSpending? **You can help us reach as many people as possible** by helping us translate OpenSpending into your language (if it isn't available in it yet). Just go to our [project page on Transifex](https://www.transifex.com/projects/p/openspending/) and offer help with translations into your language (or ask us to create your language if it isn't there already). When accepted, just jump in and start translating OpenSpending. + +### OpenSpending visualisation library + +Probably one of the bigger changes in December was what we did to OpenSpendingJS, which we use for our visualisations. We took the first official step in our move away from the *bunch-of-javascript-code* approach to versioned releases of a javascript visualisation library. + +This means that we build a version of the library and you can just drop that into your web page and be certain that it won't change in the future. This frees us from the problem of not being able to take OpenSpendingJS into new directions. Another big change this brings with it is a clearer distinction between OpenSpending core javascript and OpenSpendingJS visualisations. The previous way to add a visualisation was weird. Either you added it into the OpenSpending core visualisation builder or made it available outside it in a way developers had to grok the whole project to be able to use some visualisation. + +Now we've made it as easy as possible to add visualisation based on OpenSpending to your sites, drop in the javascript file and create a beautiful visualisation with a simple div element. Want to create a treemap for your dataset? Just do this: + +

    +Want more configurations of your treemap. You can also create these elements with a jQuery extension call: + + $('#id-of-div').treemap({/*configuration object*/}); + +This also hopefully makes it simpler to contribute new visualisations to OpenSpendingJS. Just follow how all of the other visualisations are doing things by looking at the code in *src/visualisations/*. They're built as jQuery extensions and are well commented. + +We've already made two bigger releases (0.1.0 and 0.2.0) and one bugfix release (0.2.1) so you can expect a lot of activity around these in the future. **Want to help out?** Just dive in and help us mold and improve OpenSpendingJS. One thing we want to do is clean up the older code a bit, perhaps package it up in one release and then just remove the code. We won't hit version 1.0 until at least when we've done that. + +### OpenSpending WordPress plugin + +Another thing the new approach to creating the visualisations in OpenSpendingJS as a library allows us to do is create CMS plugins for our visualisations... and that's exactly what we did. We released our WordPress plugin into the wild. We started work on it a while back but now, with release 0.5, we think it's usable to the public. + +First of all it includes the new OpenSpendingJS library (version 0.2) and we can easily update it with new releases of the OpenSpendingJS library. The other reason is that we submitted it to WordPress and got approved so we'll be adding it as an official plugin, installable from any WordPress instance in the world with a click of a button (we have yet to upload it, but that's going to happen really soon). + +The WordPress library uses the new visualisation rendering technique by creating divs with the visualisation configurations. We try to standardise the input for all visualisations to focus more on ease of use instead of configurability. If you're configuring the visualisations a lot, we believe you'll know how to add a javascript file to your page and call it directly. For some users, ease of use is more important and that's where the WordPress plugin comes in. Interested in how it works? Take a look at [the documentation](https://github.com/openspending/openspending-wordpress-plugin/blob/master/docs/user-interface.md) + +This is yet another software package we're maintaining as part of the ever-increasing arsenal of OpenSpending tools. **That's why we need your help!** Do you know PHP/WordPress? Help us improve and maintain the WordPress plugin. It's quite simple code. Go take a look at [the repository](https://github.com/openspending/openspending-wordpress-plugin). + +### Bar charts + +One thing which finally got released with the new OpenSpendingJS library after having been around for a long time in the code branch containing the new library set up is bar charts. Yes, we now have bar charts! The bar chart implementation that we had in that branch was developed to present time series only, but before releasing the first version of OpenSpendingJS they got a face lift. They can now show time series if you include time in your drilldowns but they can also show COFOG categories (with our well known icons). + +This got added into the OpenSpending WordPress plugin discussed earlier so you can easily create the bar charts now, either using OpenSpendingJS directly or the OpenSpending WordPress plugin. Yay! + +### Inflation support in visualisations + +A while back we implemented inflation support in OpenSpending. This has only been accessible to those who know about it but with the release of OpenSpendingJS version 0.2 we have added support for inflation adjustments. This is especially important when creating time series with the new bar charts since that compares data across history and each amount is valued differently. + +Inflation adjustment is done automatically in the OpenSpending WordPress plugin where we try to inflate to the value of the year before the current year. In the case where this fails, for example if there is no inflation data available (e.g. when showing budgets for future years or just if the country hasn't got good data) we don't do the inflation adjustments. + +In OpenSpending this is done using the *inflate* configuration where you define the target year you want all amounts to be inflated to. + +### Flakes + +A while back we created three issues with focus on improving the code readability and speed of OpenSpending. These issues were the results of running a python source code checker, flake8 (which is just a wrapper around other source code checkers). The issues all touched upon three different checkers, one was for pep8 compatibility (coding convention), another was for pyflakes (unneccessary imports) and the third was about McCabe's complexity (too complicated methods). + +Running these checkers on the OpenSpending code base gave us a lot of errors, i.e. areas for improvement. Near the end of the year a community member decided to start doing something about it, especially the flakes errors (is there anything better to work on than flakes in December with snow flakes all around us... well if it's snowing outside). + +There were some flake errors left unfixed because they caused test failures so we need slightly more development effort to fix them but we still fixed a lot. The community member that contributed the fixes is a python beginner and will use this to learn more about python. This is just great because this shows that we truly are the newcomer friendly community we want to be. You can join and improve your skills while you work on something that can change the world! + +**Want to contribute?** The three issues are still open for volunteers. You could for example work on the [pep8 errors](https://github.com/openspending/openspending/issues/691). Nobody has called dibs on that one. + +### Other Changes + +There were of course some other changes we did in December like swapping out a footer link to YourTopia for [Spending Stories](http://spendingstories.org/). We still don't think we've done Spending Stories enough justice on the page so we'll be looking into making it more prominent. If you haven't checked that project go to [http://spendingstories.org](http://spendingstories.org) and have a look around. + +Remember that you can help develop OpenSpending on many fronts, be it the core [OpenSpending platform](https://github.com/openspending/openspending), [OpenSpendingJS](https://github.com/openspending/openspendingjs), [Taxman](https://github.com/openspending/taxman), [the satellite site template](https://github.com/openspending/satellite-template) or any other software project in the [OpenSpending project list](https://github.com/openspending/). Just have a look at our [OpenSpending issue tracker](https://github.com/openspending/openspending/issues) or our [OpenSpendingJS issue tracker](https://github.com/openspending/openspendingjs/issues), that's where most of the issues are being tracked. Feel free though to add features you think are missing in any project and become a contributor to the OpenSpending project. + +Don't know where to start? We've highlighted some stuff on our [OpenHatch page](http://openhatch.org/projects/openspending). + +### Thanks + +**agonar**, **Neil Ashton**, **Nikesh Balami**, **Vitor Baptista**, **Mark Brough**, **Sukma Budi**, **Lucy Chambers**, **Pierre Chrzanowski**, **Iván Cruz**, **Manish Dangol**, **DarisLi**, **inxaoc**, **jbricetetka**, **Martin Keegan**, **kshitizkhanal7**, **Friedrich Lindenberg**, **Elisabetta Lombardo**, **M2M**, **Leandro Martelli**, **Alexey Medvetsky**, **michalskop**, **Randy Moore**, **Prakash Neupane**, **Nela**, **nichtsistwahr**, **nkissel**, **Olga Parkhimovich**, **Anders Pedersen**, **Rufus Pollock**, **Bansaj Pradhan**, **JanaSekaninova**, **Nick Stenning**, **rajansilwal**, and **Marco Voormolen**. + +The community contributions continue to grow so this list is getting difficult to compile and there are probably a lot more community members who have contributed somehow to OpenSpending. Just let us know and we'll add you to the list. + +Image of bar chart sketch used in this blog post is by [mattcornock](http://www.flickr.com/photos/mattcornock/) on [Flickr](http://flickr.com), released under [Creative Commons Attribution, version 2.0](http://creativecommons.org/licenses/by/2.0/). + diff --git a/examples/openspending/content/blog/2014-02-10-open-up-your-citys-finances-on-open-data-day.md b/examples/openspending/content/blog/2014-02-10-open-up-your-citys-finances-on-open-data-day.md new file mode 100644 index 00000000..37293ace --- /dev/null +++ b/examples/openspending/content/blog/2014-02-10-open-up-your-citys-finances-on-open-data-day.md @@ -0,0 +1,30 @@ +--- +redirect_from: /2014/02/open-up-your-citys-finances-on-open-data-day/ +title: Open up your city's finances on Open Data Day! +authors: +- Anders Pedersen +--- +Open Data Day is coming up on February 22nd with events happening in cities across the world. Are you interested in the recent budget passed in your city council or curious about the expenditures of your local school board? Then be sure to get involved and make use of the OpenSpending platform and community at Open Data Day.
    + +Help open up your city's finances by publishing them on OpenSpending and producing elegant visualizations that show where money goes. OpenSpending makes it super-easy to turn an Excel (or even PDF) file of local city finances into something browsable and searchable. Community members have already added more than 100 city budgets to OpenSpending and plotted them on this map . It also enables you make clear and beautiful visualisations of your local city’s finances in seconds making them understandable to everyone. + +Just follow these 3 simple steps:
    + +

    1. Find the data - locate your local city or municipal data + +

    2. Prepare the data - turn it into a single clean spreadsheet in Excel, Libreoffice or Google Docs (and then export as CSV - just “save as”) + +

    3. Upload to OpenSpending and visualize +
    +If you need more information we've got the full step-by-step guide for you even in six different languages. + +With a bit of data cleaning and a quick upload you will for example get this neat visualization of the Moscow city budget for 2014: + +
    +Will you be working on budgets or spending at the Open Data Day? +
    Let the OpenSpending community know by sharing the news on the mailing list and connect with us on the IRC freenode channel #openspending. + +To learn more about where the Open Data Day events are happening check the Open Data Day wiki. The Open Data Day wiki also offers some excellent guidance for you can organise an event in your city. + +If you need inspiration for your event check this list of exciting Open Data Day activities, which have already been planned. + diff --git a/examples/openspending/content/blog/2014-02-12-help-us-keep-the-eu-farm-subsidies-available-to-the-public.md b/examples/openspending/content/blog/2014-02-12-help-us-keep-the-eu-farm-subsidies-available-to-the-public.md new file mode 100644 index 00000000..83c30f04 --- /dev/null +++ b/examples/openspending/content/blog/2014-02-12-help-us-keep-the-eu-farm-subsidies-available-to-the-public.md @@ -0,0 +1,21 @@ +--- +redirect_from: /2014/02/help-us-keep-the-eu-farm-subsidies-available-to-the-public/ +title: Help us keep the EU farm subsidies open and accessible +authors: +- Anders Pedersen +--- +50 bn EUR for farm subsidies every year in the EU. Help OpenSpending uncover where the money goes. + +This is a post by Stefan Wehrmeyer and Anders Pedersen. + +Last year OpenSpending engaged in a partnership with FarmSubsidy.org to publish the data on payments for recipients of EU farm subsidies, officially known as the EU Common Agricultural Policy. EUR 50 bn. a year are paid in farm subsidies Most of the farm subsidies are paid directly to farmers and companies behind agricultural products. The fight to open these to the public at granular level and in machine readable format has been ongoing since 2005, when transparency advocates and journalists across the EU began demanding the data to be free. While farm subsidy spending is on the decline farm subsidies will continue to account for 38 per cent of the EU budget until 2020. Farmsubsidy.org is the only public database in Europe documenting who gets the money. + +The farm subsidy data is released annually and this year the OpenSpending community will head up the data collection on farm subsidy payment, which must be published by each of the 28 EU member states by end of April. In most member states the data will not be available as bulk download, but will need to be scraped from government websites. We need your help to make the data open and accessible to the public! + +### Help us track and scrape the farmsubsidy data! +We are there now starting to collect farmsubsidy data from across 28 member states. Stefan Wehrmeyer from Open Knowledge Foundation Germany has set up a GitHub repository with an issue for each member state: + +Check and add you self to the Farm subsidy country tracker by contributing a scraper for your country (or any other). + +To uncover the EU farm subsidies we will need help from across the OpenSpending community and beyond! Documentation on the output format and ways of integration is still forthcoming, but if you are interested, post a comment on the Github issue or to the Openspending Developer list. The upcoming Open Data Day on 22 February could be a great opportunity to tackle this in your country! + diff --git a/examples/openspending/content/blog/2014-03-11-historiasdegasto-org-is-born-to-explore-public-expenditure-in-spanish.md b/examples/openspending/content/blog/2014-03-11-historiasdegasto-org-is-born-to-explore-public-expenditure-in-spanish.md new file mode 100644 index 00000000..2527ab99 --- /dev/null +++ b/examples/openspending/content/blog/2014-03-11-historiasdegasto-org-is-born-to-explore-public-expenditure-in-spanish.md @@ -0,0 +1,22 @@ +--- +redirect_from: /2014/03/historiasdegasto-org-is-born-to-explore-public-expenditure-in-spanish/ +title: historiasdegasto.org is born to explore public expenditure in Spanish +authors: +- Concha Catalan +--- +It was bound to happen in an organization such as the Open Knowledge Foundation. People who have never met face to face have been working to adapt the successful project [spendingstories.org](http://spendingstories.org) to Spanish. The API collects and compares episodes of public expenditure which are often surprising for the general public. + +Members of [okfn.es](http://okfn.es), [openkratio.org](http://openkratio.org), and [opengov.cat](http://opengov.cat) presented [historiasdegasto.org](http://historiasdegasto.org) on the occasion of International Open Data Day at the [okioconf.es](http://okioconf.es) in Seville and in Barcelona. + +[historiasdegasto.org](http://historiasdegasto.org) is the Spanish version of spendingstories.org, created by [openspending.org](http://openspending.org), a free and open database that maps public financial transactions worldwide. To use it, you simply type an amount and click on *compare*. + +historiasdegasto.org stores references to news published on several digital media about public expenditure and automatically compares the cost of infrastructures, contracts, allocated budgets, salaries, and other. Open to citizens' contributions, the page will undoubtedly become a useful tool for the media and citizens in search of a better democracy. + +[openkratio](http://openkratio.org) (previously known as OpenData Sevilla) has been campaigning for open government and open data in society and specially in the Public Administration since 2011. + +[opengov.cat](http://opengov.cat) has advocated for transparency of the Catalan government since 2013 and has published visualizations of its intricate organization charts as well as stories about documents that are not easily accessible or publicized. + +[openspending.org](http://openspending.org) encourages everybody to open the budget of their town as openkratio did with [Sevilla Presus](http://openkratio.org/index.php/sevillapresus13-periodismo-de-datos-a-nivel-municipal/). Their handbook to know how to do that is now [available in Spanish](http://community.openspending.org/help/guide/esp/). + +Who else would like to make public expenditure open? It is our own expenditure, too. + diff --git a/examples/openspending/content/blog/2014-03-28-hackathon-odd14-barcelona.md b/examples/openspending/content/blog/2014-03-28-hackathon-odd14-barcelona.md new file mode 100644 index 00000000..711abdab --- /dev/null +++ b/examples/openspending/content/blog/2014-03-28-hackathon-odd14-barcelona.md @@ -0,0 +1,78 @@ +--- +redirect_from: /2014/03/hackathon-odd14-barcelona/ +title: 'Hackathon #ODD14 Barcelona' +authors: +- Concha Catalan +--- +On Saturday, February 22, International OpenDataDay (ODD), [opengov.cat](http://opengov.cat/ca/castellano/) coordinated a hackathon for the first time. It took place in the great working space [Makers of Barcelona](http://www.mob-barcelona.com/). + +![opengov.cat](http://opengov.cat/wp-content/uploads/2014/02/opengovcatlogo.jpeg) + +A hackathon is a meeting of journalists, programmers, and developers to work on a specific project. There were about twenty people, and we suggested *three* challenges to make different information about our public +administration and the autonomous government of [Generalitat de Catalunya](http://www.gencat.cat/) more accessible. We thought one or two would be chosen, but our attendants' skills were so great that they added a new challenge to our list, and we ended up working on all our suggestions! + +![NO-PDF](http://opengov.cat/wp-content/uploads/2014/03/NO-PDF.png) + +## Challenge 1 + +*Clean the dataset of [Generalitat budget 2014](http://www20.gencat.cat/portal/site/dadesobertes/menuitem.160770a4eab24e0b16572d32b0c0e1a0/?vgnextoid=49b19ee9acb42310VgnVCM1000000b0c1e0aRCRD&vgnextchannel=49b19ee9acb42310VgnVCM1000000b0c1e0aRCRD&vgnextfmt=detall&q=pressupostos&newLang=ca_ES) to include it on [openspending.org](https://openspending.org), the free and open database of public financial transactions, and to be able to visualize it.* + +![tweet](http://opengov.cat/wp-content/uploads/2014/03/tuit-14_02_13.png) + +> Q: Hello, when will your budget dataset be available in open data? +> +> A: We plan to publish it next week. Thank you. +> +> Q: Which day next week? Thank you. + +The budget of our autonomous government for 2014 had been released in a reusable format on [the Generalitat open data portal](http://www20.gencat.cat/portal/site/dadesobertes?newLang=en_GB) only a week before, and we had exchanged a few tweets about it even before the final version was [passed by Parliament](http://www20.gencat.cat/portal/site/economia/menuitem.9542de278d3a7ee508d1b110b0c0e1a0/?vgnextoid=64be5bccbb222410VgnVCM1000008d0c1e0aRCRD&vgnextchannel=64be5bccbb222410VgnVCM1000008d0c1e0aRCRD&vgnextfmt=detall&contentid=b3d597110beb3410VgnVCM2000009b0c1e0aRCRD) on 23 January. + +![tweet](http://opengov.cat/wp-content/uploads/2014/03/tuit-14_02_14.png) + +> – The budget is available on our open data portal. +> +> – Thank you very much for posting budget 2014. + +We're still working with openspending.org and, together with [openkratio](http://openkratio.org/), we will soon contribute to launching [historiasdegasto.org](http://openkratio.github.io/okf-spending-stories/), the Spanish version of [spendingstories.org](http://spendingstories.org). All of them are projects of the [Open Knowledge Foundation](http://barcelonalittleshell.blogspot.com.es/2014/02/nace-historiasdegastoorg-para-explorar.html). + +## Challenge 2 + +*Enter senior officials' salaries and allowances on a spreadsheet so as to have it in a reusable format.* + +![Captura de pantalla 2014-03-03 a la(s) +18.35.53](http://opengov.cat/wp-content/uploads/2014/03/Captura-de-pantalla-2014-03-03-a-las-18.35.53.png) + +Such information started to be available in the Government and President section of the [Generalitat transparency portal](http://transparencia.gencat.cat/) in PDF format last summer, with no metadata about each file. It is not easy to use, but it is better than nothing: the story [We've won the Grand Prize, +627.45 euros per month](http://opengov.cat/en/2014/02/weve-won-the-grand-prize-627-45-euros-per-month/) derived from it. + +Here in [`retribucions_README`](http://opengov.cat/wp-content/uploads/2014/03/retribucions_README.txt) you can find a technical explanation about how and with what software the documents were cleaned. + +## Challenge 3 + +*Familiarise ourselves with the [Generalitat 2014 budget dataset](http://www20.gencat.cat/portal/site/dadesobertes/menuitem.160770a4eab24e0b16572d32b0c0e1a0/?vgnextoid=49b19ee9acb42310VgnVCM1000000b0c1e0aRCRD&vgnextchannel=49b19ee9acb42310VgnVCM1000000b0c1e0aRCRD&vgnextfmt=detall&q=pressupostos&newLang=ca_ES), the law that regulates the budget, the concept of chapters, items, and others; compare it to the 2012 budget (extended to 2013), to see which concepts are no longer there and the work out increases and decreases.* + +Later I was given this [interesting link](http://virtual.eapc.cat/pluginfile.php/110865/mod_resource/content/1/gest_pressup/estructura_del_pressupost._classificacio_de_l_estat_de_despeses.html) (in Catalan) which clearly details everything, straight from the Catalan Public Administration School. + +![Captura de pantalla 2014-03-04 a la(s) +01.45.43](http://opengov.cat/wp-content/uploads/2014/03/Captura-de-pantalla-2014-03-04-a-las-01.45.43.png) + +Each budget file has around 12k rows and over 25 columns. We didn't know how to go about it, and it is not easy if you aren't an expert. As we aren't, we looked for and found people who are... and they want to help us! Meanwhile, we are reviewing basic mathematical and statistical notions. + +We are working on budget changes in different Generalitat organizations, and we hope to be able to tell you more about it soon. + +## Challenge 4 + +A girl who attended the morning presentations stated: "I can only do maps. If I can help, I'll stay for the hackathon." We quickly rummaged through [what we had on ice at opengov.cat](http://opengov.cat/ca/castellano/) and found a half-made map of the Generalitat Departaments buildings, with all the available (or non-available) contact information for each one: address, phone, webpage, email, Twitter. + +In many cases, a horrible form—one of those that asks for all your personal information but leaves no trace of your information request—replaces the contact email. A form [like this](https://ovt.gencat.cat/gsitfc/AppJava/generic/conqxsGeneric.do?webFormId=6&set-locale=ca_ES). + +We had made the map some time before, but we didn't like its format, so it remained unpublished. The new one will be better and made properly from a spreadsheet using [CartoDB](http://cartodb.com/) software. + +![CartoDB logo](http://opengov.cat/wp-content/uploads/2014/03/cartodb-logo-300x150.jpeg) + +Oscar Marín from [Outliers](http://outliers.es/) gave us a workshop about how to make maps in a recent data journalism session at the CCCB that has now been posted [here](http://www.cccb.org/ca/curs_o_conferencia-periodisme_de_dades_sessi_de_treball_ii-45371). + +If you key in "periodismo de datos" and "2013" o "2014" in the search box, you'll see the videos of all the sessions held since September 2013. (PUBLICITY - In the first session, in [September 2013](http://www.cccb.org/ca/curs_o_conferencia-periodisme_de_dades_sessi_de_treball_i-44601), we presented this project!) + +Just an addition: Thanks everybody who organised, participated, and helped (specially [@doublebyte](http:/twitter.com/doublebyte)), and *happy open data!* + diff --git a/examples/openspending/content/blog/2014-05-08-meet-openspending-version-0-13-0.md b/examples/openspending/content/blog/2014-05-08-meet-openspending-version-0-13-0.md new file mode 100644 index 00000000..aad79584 --- /dev/null +++ b/examples/openspending/content/blog/2014-05-08-meet-openspending-version-0-13-0.md @@ -0,0 +1,56 @@ +--- +redirect_from: /2014/05/meet-openspending-version-0-13-0/ +title: Meet OpenSpending version 0.13.0 +authors: +- Tryggvi Björgvinsson +--- +This is going to be a slightly technical post (and has already been posted to the developer mailing list), but still it's an important change so everyone is encouraged to read it. If you don't understand something, then that's just fine, it probably does not have anything to do with you and you can skip it. + +For a long time we have had version 0.11 of OpenSpending (since October 2011). We then for a short time had a version 2.0 alongside our 0.11 (we had not reached 2.0 so it was kind of confusing, but it snuck in with the re-theme of our docs back in January -- 2.0 came in according to the python convention while OpenSpending (version 0.11) had its own convention). + +This has all now been corrected and we redid how versions are handled (this happened about a month ago). We now do versioning as recommended by Zooko (probably best known for Tahoe-LAFS and Zooko's Triangle). This is a nice mix of the two conflicting versions we had so this confusion should not happen again. + +While making those changes and removing the erroneous 2.0 I decided it was time to bump up the version to version 0.12.0 without announcing anything per se. Versions aren't as big of a deal in our continuous deployment setup (meaning we deploy changes as soon as they're ready) so nobody is really looking at the versions. + +But that's not entirely correct. Versioning, when done properly, can help both users and new contributors (and ourselves) better understand what is happening on the project and allows us later on to introduce backwards incompatible changes in a way that we can prepare users and contributors for (this is something we should always avoid, but still a safety pin worth having). Perhaps most importantly, it helps us plan for the future with milestones. What do we want to see in version 1.0.0 or 1.5.0 etc.? + +We'll be using the Semantic versioning system, introduced by Tom Preston-Werner (who created Gravatar and is one of the founders of Github). The versioning syste is a convention had been used before by numerous projects, but never officially with explicit meaning like what Tom did. In the past it was more of a project-members-decided-what-the-numbers-between-the-dots-mean basis). + +In this versioning system we have the <major>.<minor>.<patch> where major versions are backwards incompatible, minor are compatible changes, and patches are bug fixes etc (goals along the way to our next minor/major version). + +...and with that I'm going to introduce backwards incompatible changes in a minor version by introducing version 0.13.0 (oh isn't it wonderful how you can break rules as soon as you set them). OpenSpending version probably won't update that frequently in the coming months, but as more contributors jump on board and more pull requests start pouring in we'll get closer to the open source development mantra: "release early, release often". + +USERS HEADS UP: This is the important thing for you to know. We are going to have versions and as long as the number in the middle, or the last numbers change, you'll just be seeing a better OpenSpending platform. When you see the first number change you'll have to watch out. Things you expected to work might not work. Don't worry though these first number editions (major version) will not be thrown on you just like that but we'll prepare for them and let you know well in advance so you can prepare yourselves for the change if it affects you. + +We've done a lot of changes over the past few months with a lot of help from the community, especially on the code cleanup front. We are now fully pep8 compliant and pyflakes error free (meaning the code easier to read and less unnecessary things in the code). + +We should all be very thankful for the tremendous work of Jorge C. Leitão, Randal Moore, Justin Duke, and garethpdx who have helped us clean up our code base. There's still a lot of work to be done, but thanks to Jorge, Randal, Justin and garethpdx we can all now feel it, the code can be tamed! Thanks all of you! + +In those changes I felt I had to make a substantial change to how celery worked in OpenSpending. We did a lot of weird magic to hook celery into paster as a command and we had to do this strange (unused) import: + +from openspending.command import celery + +to set some configuration variables in order for celery to work. That has now been scrapped and we have a new version of celery in OpenSpending (version 3.1.11). + +Celery is no longer managed via paster, we can now use the celery commandline tool that comes with celery to launch our workers. Instead of manually setting a lot of stuff in a config file we can now just do something like: + +celery -A openspending.tasks -p <ini-file> -l info worker + +(The -p option there is still needed to provide the pylons ini file, and is an openspending extension of celery, since other configurations, such as the database are still managed by config files). + +More information about the celery tool and how it can launch workers here: +http://docs.celeryproject.org/en/latest/userguide/workers.html + +That's the big incompatible change which sparked off version 0.13.0. This is not a user facing change so I decided against a major version, but for those of us that have a development version that's a change which is pretty important to know about. + +With this announcement of a new versioning system and the new version I think it is in order to ask the community (please reply on our mailing lists so we can have a great discussion about this): + +

      +
    • What would you like to see in version 0.14.0?
      • +
    • What would you like to see in version 1.0.0? +
      • +
    +Need help in figuring out what to suggest? Just visit our issue tracker for some ideas. + +Welcome to the new versioned OpenSpending. I hope you'll all enjoy :-) + diff --git a/examples/openspending/content/blog/2014-07-17-a-specification-for-budget-data-introducing-the-budget-data-package.md b/examples/openspending/content/blog/2014-07-17-a-specification-for-budget-data-introducing-the-budget-data-package.md new file mode 100644 index 00000000..72726d6d --- /dev/null +++ b/examples/openspending/content/blog/2014-07-17-a-specification-for-budget-data-introducing-the-budget-data-package.md @@ -0,0 +1,26 @@ +--- +redirect_from: /2014/07/a-specification-for-budget-data-introducing-the-budget-data-package/ +title: 'A specification for Budget Data: Introducing the Budget Data Package' +authors: +- Lucy Chambers +--- +*As we descend into the buzz of the biggest week of the open calendar at OKFestival, the OpenSpending team wanted to quickly drop you a line about a piece of research we have been collaborating on together with the International Budget Partnership, Omidyar Network, and Google: a draft specification for budget data.* + +As members of this community will be only too aware, the benefits of structured, machine-readable, and internationally comparable budget data have now been discussed in numerous fora. A flexible but usable global data standard would need to be developed and widely adopted to make this happen. + +Recognising the significant challenges posed by variations across national and sub-national budgets, Open Knowledge, with input and technical advice from the International Budget Partnership, Omidyar Network and Google, have begun a scoping exercise, producing an initial proposal for a draft specification (Budget Data package) which could form the basis of a global standard. + +The draft specification (Budget Data Package) can be found here: + +* [https://github.com/openspending/budget-data-package](https://github.com/openspending/budget-data-package) + +Input from the whole financial data community – publishers, investigators, citizens – is going to be necessary if this specification is going to meet users' needs and become a true standard: an open consultation session begins. + +We want your feedback! There are numerous ways to leave feedback on the specification: + +1. [Via Github issues](https://github.com/openspending/budget-data-package/issues). This is also the best way to register your interest in being part of the coalition of users who will take ownership of the project's ongoing development. +2. Coming to [OKFestival](http://2014.okfestival.org/)? Why not schedule a time to talk to us directly? The budget spec featured in the session ["Government Budgets: Joining the Dots"](http://okfestival2014.sched.org/event/5451434a7bb1a7668ea4c8d589c3872e) - and we're still available to discuss the future of the project. +3. When the dust has settled from OKFestival, we will also schedule a community call to further discuss the specification. Interested in joining? [Send us an email](mailto:lucy.chambers@okfn.org). + +We look forward to speaking to you at the festival and beyond. + diff --git a/examples/openspending/content/blog/2014-08-05-au-cameroun-la-transparence-budgetaire-village-par-village.md b/examples/openspending/content/blog/2014-08-05-au-cameroun-la-transparence-budgetaire-village-par-village.md new file mode 100644 index 00000000..4e81655e --- /dev/null +++ b/examples/openspending/content/blog/2014-08-05-au-cameroun-la-transparence-budgetaire-village-par-village.md @@ -0,0 +1,38 @@ +--- +redirect_from: /2014/08/au-cameroun-la-transparence-budgetaire-village-par-village/ +title: Au Cameroun, la transparence budgétaire, village par village +authors: +- pierre chrzanowski +--- +

    Comment un outil comme OpenSpending peut aider à mieux orienter les dépenses publiques vers les services de base au Cameroun. + +

    English version here + +

    Au Cameroun, l'exploitation des forêts rapporte beaucoup d'argent. Le bois est ainsi la troisième source d'exportation du pays, après le pétrole et le cacao. En contrepartie, chaque entreprise du secteur doit s'acquitter d'une redevance dont une partie revient aux communautés vivant en bordure des concessions. Selon la loi, cette redevance doit être allouée à des projets de développement local (électrification, accès en eau potable, écoles). Des comités communaux en ont la charge. + +Are forest revenues of Yokadouma, a council in the East region of Cameroon, invested in basic services? Image credit: Courtesy of Paolo Scoppola - www.paoloscoppola.com + +

    Dans un pays où l'accès aux services de base reste encore largement insuffisant pour les populations rurales, la redevance d'exploitation forestière devrait donc constituer un levier important pour le développement local. Mais quel est son impact réel ? Difficile de savoir quand les informations sur les investissements publics ne sont pas accessibles. Les populations bénéficiaires devraient pourtant pouvoir controler si la redevance est effectivement perçue, redistribuée et correctement investie. C'est tout l'enjeu de la transparence budgétaire. + +

    Ainsi, dans le cadre du programme de la Banque Mondiale, Initiative pour la Transparence Budgétaire au Cameroun, l’équipe OpenSpending a développé en 2012 le site cameroon.openspending.org, une application dédiée à l'exploration et au suivi des budgets. Le site permet de naviguer à travers les différent niveaux administratifs du pays et de visualiser les dépenses publiques plannifiées et exécutées qui s'y rapportent. + +

    Chacun peut contribuer à la base de données et ajouter les informations budgétaires de sa commune. Cela requiert cependant certaines compétences techniques. C'est pourquoi une seconde mission a eu lieu en juin 2014 pour former l'association ASSOAL et ses partenaires. L'objectif était de transmettre aux participants les compétences et outils nécessaires pour maîtriser l'ensemble de la procédure, de la collecte à la publication en ligne des données budgétaires des communes. + +

    The aim of the workshop was to transform budget papers into easy-to-understand online information. + +

    La démarche d'ASSOAL vise à promouvoir et démontrer les bienfaits de la transparence sur le développement local. Les informations que l'association publie sont accessibles à tout citoyen connecté et peuvent également être réutilisées pour la préparation des budgets participatifs, ou pour la réalisation de fresques murales ou émissions de radios citoyennes. À terme, l'association souhaite pouvoir mettre en ligne et explorer l'ensemble des budgets planifiés et éxecutés des 336 communes du pays. Cela nécessitera une plus forte collaboration avec les instances publiques, comme le FEICOM chargé des investissements intercommunaux, et la mise en place de nouvelles procédures de gestion des données budgétaires. La mission a ainsi été l'occasion d'insister sur l'importance de disposer de données budgétaires dans des formats numériques réutilisables et non dans des formats papiers ou PDF. + +

    Pour l'heure, les membres d'ASSOAL saisissent manuellement sur leur ordinateur les données contenues dans les budgets papiers qu'ils collectent auprès des communes. Un travail fastidieux donc, et source d'erreurs, mais qui à défaut de budgets disponibles au format numérique reste la seule solution pour pouvoir ensuite publier les données en ligne. + +

    L'atelier a permis aux participants d'apprendre à nettoyer, corriger et formater les données tabulées en utilisant des outils simples tels que Excel, LibreOffice ou encore OpenRefine. Ils ont également pu se former à la collaboration et au partage de données en ligne grâce à des outils comme Etherpad ou Datahub.io. L'atelier a été l'occasion d'améliorer la procédure de saisie des données budgétaires, en adoptant une structure de données plus conforme à la nomenclature réelle et en s'inspirant des spécifications du nouveau Budget Data Package de OpenSpending. À la suite des 4 jours de formation, l'équipe a été capable de traiter et publier 40 budgets en un mois. L'objectif est de doubler ce chiffre pour fin 2014. + +

    Explore how much a council earns from forest tax and how much it spends in basic services on cameroon.openspending.org. + +

    Grâce aux efforts de la societé civile et à l'usage d'outils comme OpenSpending, la transparence budgétaire progresse au Cameroun, mais essentiellement au niveau des communes. C'est un premier pas encourageant qui, il faut l'espérer, incitera le gouvernement à faire de même au niveau national. L'enjeu ici est bien d'améliorer le niveau de vie des populations en leur permettant de contribuer à leur propre développement. Ces dernières devraient donc être en mesure de connaître leurs droits, être capables de suivre les décisions de ceux qui les représentent et les obliger à rendre des comptes. La transparence et la compréhension des budgets sont donc ici essentielles, tout comme le sont l'accès aux textes de loi et, dans ce cas précis, l'accès aux informations sur les exploitations forestières du pays. + +  + +  + +  + diff --git a/examples/openspending/content/blog/2014-08-05-in-cameroon-budget-transparency-one-council-at-a-time-2.md b/examples/openspending/content/blog/2014-08-05-in-cameroon-budget-transparency-one-council-at-a-time-2.md new file mode 100644 index 00000000..ccb92642 --- /dev/null +++ b/examples/openspending/content/blog/2014-08-05-in-cameroon-budget-transparency-one-council-at-a-time-2.md @@ -0,0 +1,40 @@ +--- +redirect_from: /2014/08/in-cameroon-budget-transparency-one-council-at-a-time-2/ +title: In Cameroon, budget transparency one council at a time +authors: +- pierre chrzanowski +--- +How a tool like OpenSpending can help to better channel public spending into basic services in Cameroon. + +Version française + +

    In Cameroon, forest exploitation yield lots of money. Wood is indeed the third largest source of exports of the country, following oil and cocoa. In return, every logging company must pay a tax whose a part goes to local communities living near the concession. By law, this money must be allocated to local development projects (schools, electrification, water access). Dedicated local committees are responsible for the budgets. + +Are forest revenues of Yokadouma, a council in the East region of Cameroon, invested in basic services? Image credit: Courtesy of Paolo Scoppola - www.paoloscoppola.com + +

    In a country where access to basic services is still insufficient for many rural populations, the forest tax appears as a potential leverage for local development. But what is it real impact? This is difficult to say when you cannot access even basic council's budget data. Yet, beneficiaries, the citizen, should be able to check if taxes are effectively perceived, redistributed and efficiently allocated. This is what budget transparency is all about. + +

    As part of World Bank's Budget Transparency Initiative in Cameroon, the OpenSpending team developed in 2012 the website cameroon.openspending.org, a tool to explore and follow budget data. The website enables everyone to navigate through the different administrative levels of the country and to visualize related planned and executed public spending. + +

    Anyone can contribute to the database and publish budget data from its council. The technical requirements for contribution were however still significant. Therefore Open Knowledge teamed up with the local civil society organisation ASSOAL and its partners in June 2014 to make contribution and dissemination of local budget data more accessible. The objective was to transfer skills and tools between participants and enable ASSOAL and local partners to steer the entire process: from data collection to online publication of councils budget data. + +

    The aim of the workshop was to transform budget papers into easy-to-understand online information. + +

    ASSOAL aims to promote and demonstrate the benefits of transparency on local development. Information published by the NGO are available to every connected citizen but can also be used for participatory budgets, citizen radio programs or even murals to inform every citizen. Eventually, ASSOAL aims to publish every budget of the 336 councils of Cameroon online. This will require collaboration with public authorities, such as  Cameroon’s municipal bank (FEICOM), in charge of cross-councils investments, but also new budget data methodologies. The workshop with ASSOAL was indeed an opportunity to discuss the importance of getting budget data online in digital and reusable format - not only in paper or PDF. + +

    Currently, ASSOAL's members are entering or transcribing budget data manually on their computers from paper versions. This is a tedious process, and also a great source of errors, but the only current way to get the data released openly on websites such as OpenSpending. + +

    The workshop enabled the participants to learn how to clean, correct, and format datasets by using simple tools such as Excel, LibreOffice or OpenRefine. Participants were also able to train on how to collaborate and share data online with applications like Etherpad or Datahub.io. The workshop was also an opportunity to review the process of data collection. At the end, the team revised its methodology by adopting a new data model and using some OpenSpending's Budget Data Package specifications. Following the 4 days training, the participants were able to publish 40 budgets in one month. The target is to double this number by the end of the year. + +

    Explore how much a council earns from forest tax and how much it spends in basic services on cameroon.openspending.org. + +

    Thanks to the efforts of the civil society and the use of tools like OpenSpending, budget transparency is slowly taking off in Cameroon from bottom up, starting at council level. This is an encouraging effort and we hope this will lead the Government to do the same with national Public Investment data. The population should be able to know their rights, follow the decisions of the ones who represent them and hold them accountable. Budget transparency is essential here, as is the right to access legal information, and in this precise case, the right to access information on forest exploitations and revenues. In the challenge of improving the standard of living for communities across councils access to budget information is key for enabling citizens to contribute and decide on the development. + +  + +  + +  + +  + diff --git a/examples/openspending/content/blog/2014-10-15-how-to-create-a-budget-data-package.md b/examples/openspending/content/blog/2014-10-15-how-to-create-a-budget-data-package.md new file mode 100644 index 00000000..200d9eab --- /dev/null +++ b/examples/openspending/content/blog/2014-10-15-how-to-create-a-budget-data-package.md @@ -0,0 +1,86 @@ +--- +redirect_from: /2014/10/how-to-create-a-budget-data-package/ +title: How to create a budget data package +authors: +- Neil Ashton +--- + +This tutorial will show you how to create a [budget data package](https://github.com/openspending/budget-data-package/blob/master/specification.md) from a (relatively clean) spreadsheet dataset by walking you through the process of converting the [Armenian budget](http://wbi.worldbank.org/boost/country/armenia) from the [Open Budgets Portal](http://wbi.worldbank.org/boost/). + +## Getting started + +The Armenia BOOST government expenditure database contains planned, adjusted, and executed expenditures covering the years 2006 to 2012. It is coded with rich classification systems, including COFOG functional categories. This makes it perfect as an example budget data package dataset. + +To download the Armenia dataset, go to the [Armenia BOOST Socrata instance](https://databox.worldbank.org/dataset/Armenia/4bk3-zxmf) and export the dataset as a CSV. + +Export as CSV + +This dataset now needs to be cleaned and processed. To do this, we will use [OpenRefine](http://openrefine.org/). + +## Processing data: splitting fields + +Before we can use the Armenia dataset in Budget Data Package, a few properties need to be fixed. The easiest of these processing steps is splitting up numerical IDs and human-readable text, which the source dataset combines together in single fields. This is easy to do with OpenRefine. + +To split up a column like this, click the arrow next to the column name, select **Edit column**, and click **Split into several columns**. + +Splitting columns + +Each column is in the format "1234 Description", where 1234 is the ID. We can therefore extract the numerical ID by splitting the column up on spaces and limiting the resulting number of columns to 2. + +Splitting on spaces + +Do this with each column that combines a numerical ID with a text description. + +At this point, you can also rename the columns in the dataset to match the budget data package specification. The columns "Administrative Classification (Level 1) -- Agency 1" and "Administrative Classification (Level 1) -- Agency 2" resulting from splitting up "Administrative Classification (Level 1) -- Agency" can be renamed "adminID" and "admin", for example. + +## Processing data: programming step + +Once you've renamed the columns appropriately, you can move on to performing three more complex processing steps: splitting up the dataset by year and status; adding unique IDs to data rows; and fixing the COFOG values. + +A simple Python script that performs these processing steps is available here: + +* [Armenia data processing script](https://gist.github.com/nmashton/442cea7f852ee92c343e) + +### Splitting up files + +Budget Data Package datasets represent a single fiscal year at a single stage in the budget cycle. + +Our source dataset combines many fiscal years, and every row in the dataset also contains money values from three separate stages in the cycle. + +It's therefore necessary to split up the source dataset into several files. This involves two steps: + +1. Turning each row into three rows, one for each money value. +2. Splitting up the set of rows by year. + +Lines 15-45 of the [Armenia data processing script](https://gist.github.com/nmashton/442cea7f852ee92c343e#file-armenia-processing-py-L15) carry out these two transformations. + +### Adding unique IDs + +Each row of data in a Budget Data Package must have a unique identifier in its `id` field. + +This is easy to do; just append an `id` field to the header row of each dataset, then add a unique value to every data row. + +Lines 48-67 of the [Armenia data processing script](https://gist.github.com/nmashton/442cea7f852ee92c343e#file-armenia-processing-py-L48) add IDs in this way. Here, unique UUIDs are generated and added to the data. + +### Fixing COFOG values + +Budget Data Package datasets need to have well-formed COFOG values in the `cofog` column. Here, "well-formed" means that the values comply with [the COFOG standard](http://data.okfn.org/data/core/cofog#resource-cofog). + +Our source dataset has a (mostly) COFOG-compatible functional classification system, but it formats its COFOG codes in an idiosyncratic way. It's necessary to transform these codes from values like "010101" to values like "01.1.1" for compliance. + +Lines 69-126 of the [Armenia data processing script](https://gist.github.com/nmashton/442cea7f852ee92c343e#file-armenia-processing-py-L69) fix all the COFOG values. They repair the existing COFOG-compatible functional classification codes, and they also add a new `cofog` column for good measure. + +## Adding metadata + +Once all datasets have been processed and made ready for Budget Data Package, they need to be wrapped up with a metadata file. + +A [sample metadata file](https://gist.github.com/nmashton/442cea7f852ee92c343e#file-armenia-json) for the Armenia BOOST dataset, as prepared in the last section, is available as a Gist. This file illustrates several crucial features of BDP metadata: + +* Metadata for the data package itself goes in the outermost object. This includes the data package's name (a URL-compatible string), title (a human-readable name), description (a prose description of the package), and version (a version number for the release). +* Each CSV included in the package needs its own metadata in the `resources` field of the package metadata. (See the Budget Data Package specification for details about what needs to go here.) + +You can see from looking at the metadata that it is mostly repetitious and predictable—time-consuming to create by hand, but not too challenging! + +## Wrapping up + +Once the metadata file for the budget data package has been created and saved alongside the processed datasets, the budget data package has been created. You can now serve it up from whatever platform you prefer. diff --git a/examples/openspending/content/blog/2015-03-20-presenting-public-finance-just-got-easier.md b/examples/openspending/content/blog/2015-03-20-presenting-public-finance-just-got-easier.md new file mode 100644 index 00000000..ed0a9c84 --- /dev/null +++ b/examples/openspending/content/blog/2015-03-20-presenting-public-finance-just-got-easier.md @@ -0,0 +1,26 @@ +--- +redirect_from: /2015/03/presenting-public-finance-just-got-easier/ +title: Presenting public finance just got easier +authors: +- Tryggvi Björgvinsson +--- +This blog post is cross-posted from the CKAN blog. + +mexico_ckan_openspending + +CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. + +One of the new features of this release is the ability to create extensions that get called before and after a new file is uploaded, updated, or deleted on a CKAN instance. + +This may not sound like a major improvement  but it creates a lot of new opportunities. Now it's possible to analyse the files (which are called resources in CKAN) and take them to new uses based on that analysis. To showcase how this works, Open Knowledge in collaboration with the Mexican government, the World Bank (via Partnership for Open Data), and the OpenSpending project have created a new CKAN extension which uses this new feature. + +It's actually two extensions. One, called ckanext-budgets listens for creation and updates of resources (i.e. files) in CKAN and when that happens the extension analyses the resource to see if it conforms to the data file part of the Budget Data Package specification. The budget data package specification is a relatively new specification for budget publications, designed for comparability, flexibility, and simplicity. It's similar to data packages in that it provides metadata around simple tabular files, like a csv file. If the csv file (a resource in CKAN) conforms to the specification (i.e. the columns have the correct titles), then the extension automatically creates the Budget Data Package metadata based on the CKAN resource data and makes the complete Budget Data Package available. + +It might sound very technical, but it really is very simple. You add or update a csv file resource in CKAN and it automatically checks if it contains budget data in order to publish it on a standardised form. In other words, CKAN can now automatically produce standardised budget resources which make integration with other systems a lot easier. + +The second extension, called ckanext-openspending, shows how easy such an integration around standardised data is. The extension takes the published Budget Data Packages and automatically sends it to OpenSpending. From there OpenSpending does its own thing, analyses the data, aggregates it and makes it very easy to use for those who use OpenSpending's visualisation library. + +So thanks to a perhaps seemingly insignificant extension feature in CKAN 2.3, getting beautiful and understandable visualisations of budget spreadsheets is now only an upload to a CKAN instance away (and can only get easier as the two extensions improve). + +To learn even more, see this report about the CKAN and OpenSpending integration efforts. + diff --git a/examples/openspending/content/blog/2015-04-15-announcing-a-new-architectural-roadmap-for-openspending.md b/examples/openspending/content/blog/2015-04-15-announcing-a-new-architectural-roadmap-for-openspending.md new file mode 100644 index 00000000..737b69bc --- /dev/null +++ b/examples/openspending/content/blog/2015-04-15-announcing-a-new-architectural-roadmap-for-openspending.md @@ -0,0 +1,33 @@ +--- +redirect_from: /2015/04/announcing-a-new-architectural-roadmap-for-openspending/ +title: Announcing A New Architectural Roadmap for OpenSpending +authors: +- Meg Foulkes +--- + + + + 01-tech-overview-polaroid + + +At the 2015 Open Data Day a proposal for a new vision for the approach +and architecture of OpenSpending was approved. It opens up an exciting +opportunity for open budget initiatives around the world to work more +closely together, whilst remaining independent. In a nutshell: + +> We want to centralize data but decentralize 'presentation' (views). + +What does this mean? It means an OpenSpending ecosystem of smaller +pieces, more loosely joined. It means an OpenSpending to which it is +easier to contribute, in which it is easier to connect your solution +to the global budget data initiative. It means an OpenSpending that is +easier to improve; instead of one monolithic codebase that nobody +wants to touch, it means smaller , more accessible projects that we +hope will inspire your participation. In short, it’s an empowering of +the awesome OpenSpending community which lies at the heart of +everything we do. + +You can read the proposal that was accepted in full here. Many +thanks to Tryggvi and Rufus for working so hard on this. Now over to +you! What do you think? We’d love to hear your thoughts. diff --git a/examples/openspending/content/blog/2015-05-05-openspending-community-hangout-revised-date-thursday-14th-may.md b/examples/openspending/content/blog/2015-05-05-openspending-community-hangout-revised-date-thursday-14th-may.md new file mode 100644 index 00000000..9243f6e1 --- /dev/null +++ b/examples/openspending/content/blog/2015-05-05-openspending-community-hangout-revised-date-thursday-14th-may.md @@ -0,0 +1,14 @@ +--- +redirect_from: /2015/05/openspending-community-hangout-revised-date-thursday-14th-may/ +title: 'OpenSpending Community Hangout: Revised Date - Thursday 14th May' +authors: +- Meg Foulkes +--- +Our next community hangout will be on Thursday 14th May at 17:00 BST. If you haven't signed up yet and you'd like to come along, please add your name to the OpenSpending Hangout Scratchpad. + +We're looking for volunteers to give lightning talks and short updates from around the community - if you are interested in giving one on your project or work please just adda note next to your name on the Scratchpad, as above. Anders has kindly volunteered to give a short talk on extractives data but it'd be great to hear from more of you! + +The hangout will be via Google Hangout and we'll circulate the link nearer the date. + +Looking forward to seeing you there! + diff --git a/examples/openspending/content/blog/2015-07-09-openspending-community-new-home.md b/examples/openspending/content/blog/2015-07-09-openspending-community-new-home.md new file mode 100644 index 00000000..64cf6fc1 --- /dev/null +++ b/examples/openspending/content/blog/2015-07-09-openspending-community-new-home.md @@ -0,0 +1,38 @@ +--- +title: "OpenSpending Community's New Home" +authors: +- Dan Fowler +--- + +![Migration](/img/blog/2015/07/migration.png) + +Welcome to the new OpenSpending Community site! This site is now +built using [Jekyll](http://jekyllrb.com/) and hosted on +[GitHub Pages](https://pages.github.com/). What does this mean for +you? + +> It means that going forward, it should be easier to tweak, update, + and contribute to the site to better fit the needs of the + OpenSpending community (e.g. you). + +Until recently, this site was hosted using +[Wordpress](https://wordpress.org/). While Wordpress served its +purpose well, it lacked the degree of flexibility we desired in a +community-oriented site. Ultimately, we +[chose](https://discuss.okfn.org/t/openspending-community-website-migration-2015/330) +to migrate to GitHub Pages because it gives us the best mix of: + +- Ease of deployment and maintenance +- Ease of editing (can do one-click editing and see changes appear) +- Auditable / moderatable +- Configurability and themability + +In the process of migrating this site, we also reorganized some of its +content, so please let us know if something is broken or not where it +is supposed to be by creating an issue on the GitHub +[repository](https://github.com/openspending/dotorg/issues/) that hosts this +site. + +Contributing is easy and, of course, highly encouraged! Guidelines +for contributing content to the site are available +[here](http://community.openspending.org/meta/contribute/). diff --git a/examples/openspending/content/blog/2015-11-30-openspending-next-spendb.md b/examples/openspending/content/blog/2015-11-30-openspending-next-spendb.md new file mode 100644 index 00000000..a02b54e2 --- /dev/null +++ b/examples/openspending/content/blog/2015-11-30-openspending-next-spendb.md @@ -0,0 +1,30 @@ +--- +authors: +- Friedrich Lindenberg +- Rufus Pollock +title: "OpenSpending Next + SpenDB = <3" +--- + +*This is a post by [Friedrich Lindenberg](https://discuss.okfn.org/users/pudo/activity) and [Rufus Pollock](https://discuss.okfn.org/users/rufuspollock/activity). Friedrich and Rufus have both been major and ongoing contributors to the OpenSpending project. Friedrich was the creator of [OffenerHaushalt](http://db.offenerhaushalt.de/), the technical lead of OpenSpending 2011-2013, and the developer of [SpenDB](http://spendb.org/). Rufus is Founder of [Open Knowledge](https://okfn.org/) and original project lead on [WhereDoesMyMoneyGo](http://wheredoesmymoneygo.org/) and OpenSpending and has continued to play an active role.* + +![Next + SpenDB](/img/next-and-spendb.png) + +This post sets out some of our shared thinking about OpenSpending and our shared commitment to collaborate together and with the community going forwards. We are committed to creating a project that is open, impactful and exciting. There is huge need for OpenSpending at both a technical and a community level. We're excited about both achievements to date and the plans for the future. We want to do everything *we* can to contribute to that future. + +Since OpenSpending evolved from the "[Where Does My Money Go?](http://wheredoesmymoneygo.org/)" project in late 2011, data-driven approaches to fiscal analysis have made incredible progress. + +In areas like international aid, payments around extractives and procurement, governments have adopted data publication as a path towards transparency and accountability. Amazing groups like [Global Witness](https://www.globalwitness.org/en-gb/) and others have adopted data as a means to explore corruption, social injustice and failed government policies. + +At the same time, there's a still a large, unsatisfied need to the right tools to do this work: open source software that allows activists, journalists, and policy wonks to easily explore fiscal data, share it with others and present their findings. OpenSpending, since it's beginning, has been focussed on creating such tools. + +Going forward, we're excited to see both branches of OpenSpending develop: on the one hand [the platform and suite of open source tools and standards](https://github.com/openspending/); on the other hand, [the community and the conversations](https://discuss.okfn.org/c/openspending) that have always made OpenSpending a special and welcoming place for anyone interested in fiscal transparency. + +Within the community, our goal is to make OpenSpending a meeting point for anyone interested in understanding how government manages and distributes public money. This is not a conversation about tools and widgets, but one about how open data can be used in the service of public education, accountability and wider engagement with public finances. + +At the same time, this is an exciting moment for the technologies surrounding OpenSpending: the [main software platform](http://openspending.org/) is about to get a [major upgrade with a more flexible architecture, a new standard and the latest technologies](/next/). This will provide the platform with a new and much improved level of usability, scalability and robustness. + +The [Fiscal Data Package (FDP)](http://fiscal.dataprotocols.org/) is a light-weight, descriptive data model that can be used to describe and compare a wide variety of fiscal data - it will act as a backbone to connect many of the applications in the OpenSpending technology stack. + +We're also excited to combine components from the [OpenSpending Next](/next/) and [SpenDB](http://spendb.org/) development paths of OpenSpending into a group of lightweight tools that can be used to drive both the existing OS budget sites (like [Spending.jp](http://spending.jp/), [OffenerHaushalt](http://db.offenerhaushalt.de/), [Cameroon's Explorateur budgétaire](http://cameroon.openspending.org/fr/), etc.), but also lend themselves to more analytical exploration of the data and custom visualisations in the context of advocacy, data journalism and policy analysis. + +Lastly, we want to talk about our own roles. We've both contributed much to the development of OpenSpending and the community around it. And we intend to keep doing so. At the same time, the project has grown and neither of us are working on OpenSpending full-time. As such, we want to define our roles more clearly. In particular, technical leadership of the project now sits with the technical team and, ultimately, with the current technical lead [Paul Walsh](https://discuss.okfn.org/users/pwalsh/activity). We will both be part of the technical team and Friedrich will be taking a leading role in the development of the visualization component whilst Rufus will be helping oversee the development of [Fiscal Data Package](http://fiscal.dataprotocols.org/). diff --git a/examples/openspending/content/blog/2016-01-20-moving-to-fiscal-data-package-0-3-0.md b/examples/openspending/content/blog/2016-01-20-moving-to-fiscal-data-package-0-3-0.md new file mode 100644 index 00000000..9b31a9b4 --- /dev/null +++ b/examples/openspending/content/blog/2016-01-20-moving-to-fiscal-data-package-0-3-0.md @@ -0,0 +1,13 @@ +--- +title: Moving to Fiscal Data Package 0.3.0 +authors: +- Dan Fowler +--- + +After hovering at alpha for a few months, we're looking to move the [Fiscal Data Package](http://fiscal.dataprotocols.org/) specification to version 0.3.0 by 31 January 2016. + +**What does this mean?** + +We would like to request interested parties read the [current spec](http://fiscal.dataprotocols.org/) and raise any comments they have on the [issue tracker](https://github.com/openspending/fiscal-data-package/issues). We are looking to solve the most pressing issues for a 0.3.0 release. Please note, that even after the deadline, the specification will still be evolving. As such, comments that don't make the 0.3.0 release could possibly be included in a future release. + +In parallel with this, we are seeking to define and refine a process for reviewing and accepting contributions. Please see the following pull request for details: [#122](https://github.com/openspending/fiscal-data-package/pull/122/files). diff --git a/examples/openspending/content/blog/2016-06-14-announcing-the-Alpha-Version-of-Open-Spending-Next.md b/examples/openspending/content/blog/2016-06-14-announcing-the-Alpha-Version-of-Open-Spending-Next.md new file mode 100644 index 00000000..dfe9d9c2 --- /dev/null +++ b/examples/openspending/content/blog/2016-06-14-announcing-the-Alpha-Version-of-Open-Spending-Next.md @@ -0,0 +1,65 @@ +--- +title: "Announcing the Alpha Version of OpenSpending Next" +authors: +- The OpenSpending Team +--- + +We’re happy to announce that a new, improved version of OpenSpending is now being launched as an Alpha Version. OpenSpending Next provides a set of tools enabling users to visualise, analyse and publish budget and spending data - all based on the open Fiscal Data Package standard. + +OpenSpending Next is designed, developed and maintained by Open Knowledge International and, as an open source project, also reflects the very valuable contributions of an active, passionate and committed [community](http://community.openspending.org/get-involved/). + +## [next.openspending.org](http://next.openspending.org) + +OpenSpending Next fills a gap in the lack of open tools that offer easy budget data publication, visualisation and analytics for individuals and organisations working with fiscal data, as well as provide a data model standard. + +The aim of the project is to support the international open data community by offering a platform that makes it easier for civil society organisations, activists and journalists to see how and where public money is being spent. OpenSpending is also targeted to governments and any regional, local administration or municipality level to increase fiscal transparency. + +The [Fiscal Data Package standard](http://fiscal.dataprotocols.org/spec/) unveils spending data and fiscal documents at all stages of the budget process by defining a common data model. + +[OpenSpending Next](http://next.openspending.org/) Alpha Release comes with core tools, including a fully working API: + +* **OS Packager** enables users to map any fiscal data against the standard, and create a Fiscal Data Package from CSV files with a simplified workflow. This package can be uploaded to OpenSpending API with just a few clicks. +* **OS Viewer** lets users create and customise visualisations based on their budget and spending data. Treemaps, pie charts, bar charts, geolocation, line charts, tables, you name it! +* **OS Explorer** allows discovering and exploring our rich collection of fiscal data sets. +* **OS DataMine**: A rich analytical tool based on [re:dash](http://redash.io/) that allows performing deep queries into fiscal data. +* **OS API**: This V3 API is almost entirely based on the excellent [Babbage](https://github.com/openspending/babbage). This API also features a fully backwards compatible implementation of the OpenSpending V2 API for legacy apps that depend on it. + +![OS-Next](http://blog.okfn.org/wp-content/uploads/2016/06/Capture-d’écran-2016-06-13-à-17.40.16-1024x626.png) + +Since we started developing OpenSpending Next, we’ve been working in the following general areas: + +* Collaborating with our partners [GIFT](http://www.fiscaltransparency.net/) and the [BOOST World Bank team](http://wbi.worldbank.org/boost/boost-initiative) to develop the [Fiscal Data Package](http://fiscal.dataprotocols.org/spec/). We’ve trialled and integrated datasets from different Boost countries with the Fiscal Data Package such as: + * [Moldova](http://next.openspending.org/viewer/boost:boost-moldova-2005-2014?measure=adjusted.sum&order=adjusted.sum%7Cdesc&visualizations%5B%5D=Treemap&groups%5B%5D=location_2.location) + * [Tunisia](http://next.openspending.org/viewer/boost:boost-tunisia-2008-2014?measure=PAYE.sum&order=PAYE.sum%7Cdesc&visualizations%5B%5D=Treemap&groups%5B%5D=administrative_classification_2.ADMIN1&rows%5B%5D=activity_2.PROG&columns%5B%5D=date_2.YEAR) + * [Peru](http://next.openspending.org/viewer/boost:boost-peru-2012-2014?measure=Executed.sum&order=Executed.sum%7Cdesc&visualizations%5B%5D=Treemap&groups%5B%5D=functional_classification_2.Function1&rows%5B%5D=activity_Program1.Program1&columns%5B%5D=date_2.Year) +* Collaborating in the [OpenBudgets.eu](http://openbudgets.eu/) project, together with [Open Knowledge Germany](https://okfn.de/), [Open Knowledge Greece](http://okfn.gr/), [J++](http://www.jplusplus.org/en), [Transparency International EU](http://www.transparencyinternational.eu/), [University of Prague](http://www.vse.cz/) , [University of Bonn](https://www.uni-bonn.de/) and [Civio](http://www.civio.es/) to build a platform that improves transparency and opens the black box of European structural funds spending data. + Read about our latest news and research topics on our [blog.](http://openbudgets.eu/blog/) - we're currently investigating EU funds and how the EU money flows. + +This work has been also funded by [Adessium Foundation](http://www.adessium.org/) as part of OpenSpending EU, a project that aims at making EU Budgets and Spending data, as well as EU Structural funds accessible to citizens, researchers, journalists and policy-makers. + +## How can you contribute to OpenSpending? + +There are so many ways to contribute to OpenSpending! + +OpenSpending is guided by community principles. Anyone interested in financial data can contribute. We’re friendly to newcomers and old hands alike – everyone was new once, and we value all levels of experience. + +* Do you have an experience working with budget data? Don’t wait any longer and start using the tool, we would welcome any feedback about your impressions on the new version and how it can be improved to serve your needs. + +* If you’re a budget activist – look at the available data and upload the newest data for your country or municipality, in order to make it available for the larger community. + +* Do you run a budget data or fiscal transparency portals? You will be able to use OpenSpending data visualizations and embed them in your portal. + +* Are you a government representative? We can’t wait for you to adopt the Fiscal Data Package as an international standard. + +* Are you a software developer, a data scientist or analyst? Do you have knowledge in data visualisation, data architecture, play with Python or D3? We are open to your suggestions and contributions, don’t wait any longer and take a look at the [technical documentation](http://docs.openspending.org/en/latest/developers) + +All material created within the project is open data and open content built with open tools. + +## How to reach us? +Nothing is more easy to contact us and get in touch, the team is available via these channels: + +1. In the [OpenSpending discussion forum](https://discuss.okfn.org/c/openspending) +2. On Gitter.im in the [OpenSpending chat room](https://gitter.im/openspending/chat) +3. On the [OpenSpending issue tracker](https://github.com/openspending/openspending/issues) + +Expect a lot of bugs, and a lot of updates and stories! But for now, we can’t wait to get your feedback! [START!](http://next.openspending.org/) diff --git a/examples/openspending/content/help/aggregate.md b/examples/openspending/content/help/aggregate.md new file mode 100644 index 00000000..399efc2c --- /dev/null +++ b/examples/openspending/content/help/aggregate.md @@ -0,0 +1,153 @@ +--- +section: help +lead: true +title: Aggregate API +authors: +- Neil Ashton +--- +The data source used to drive visualizations is the Aggregate API. It +can be used to flexibly generate aggregated views of the data by +applying filters and grouping criteria. + +This API is heavily based on OLAP concepts, and the documentation assumes +you know [how we store data](../../help/guide/en/data-model/). + +#### Basic call and parameters + + GET /api/2/aggregate?dataset= + +Calls will return aggregation results as JSON. If no arguments other than the +dataset are given, the whole cube is aggregated. The following parameters are supported: + +* ``dataset`` (required) + The dataset name to query. + +* ``measure`` + The name of the measure over which aggregation will be performed. Defaults to + ``amount``. + + Multiple measures in a single query are supported, separated by a pipe character: + ``measure=amount|budget`` (sums up the amount measure *and* the budget measure). + +* ``cut`` + Filter the entries to use only a part of the cube. Only cells matching all the + criteria given will be used. With ``cut=time.year:2009``, you can filter for an + attribute value. + + Multiple filters can be given separated by a pipe character: + ``cut=time.year:2009|category.name:health``. If two different filters are applied + to the same attribute, the query will include both results: + ``cut=time.year:2009|time.year:2010`` The dimensions you use for cut will be part + of the returned result. + +* ``drilldown`` + Dimension to be drilled down to. Each drilldown will split the result set to create + a distinct result (cell) for each value of the dimension or attribute in + ``drilldown``. + + For example, ``drilldown=time.year`` will return all entries in the source data + broken down by year. Multiple drilldowns can be combined: ``drilldown=year|category`` + will return one cell for each year/category combination. + +* ``page`` + Page number for paginated results. Defaults to ``1``. + +* ``pagesize`` + Size of a page for paginated results. Defaults to ``10000``. + +* ``order`` + List of attributes to be ordered by as a combination of ``criterion:dir`` + pairs. The indicated direction is either ``asc`` for ascending order + or ``desc`` for descending order. For example, ``order=year:asc|category:asc`` + sorts by year and then by category name. + +The API itself is inspired by [DataBrewery Cubes](http://packages.python.org/cubes/server.html#api), +with which we aim to be compatible. At the moment, we only implement the ``aggregate`` call of +this API and do not support hierarchical dimension queries in the same way. + +#### Result format + +The result will contain two keys, ``summary`` and ``drilldown``. The ``summary`` +represents an aggregation of the whole cuboid specified in the cut. The +amount given is the sum of all drilldowns. + +The ``drilldown`` contains a cell for each value of each drilled-down +dimension. Cells include the values of any attributes or dimensions +which served as drilldown criteria, as well as the ``cut`` attributes. + + { + "drilldown": [ + { + "volume": { + "name": "section-i", + "label": "PARLIAMENT" + }, + "amount": 267770600.0, + "num_entries": 46 + }, + { + "volume": { + "color": "#FF8C00", + "name": "section-ii", + "label": "COUNCIL" + }, + "amount": 705435934.0, + "num_entries": 26 + }, + ], + "summary": { + "amount": 973206534.0, + "num_drilldowns": 2, + "num_entries": 72 + } + } + +JSON is the default format but results of the aggregation can also be downloaded as a csv file. Just add ``format=csv`` to the URL parameters to fetch them as a csv file. + +#### Example: Where Does My Money Go? + +To highlight the use of this API, let's look at the UK Country +Regional Analysis dataset. This is a high-level survey of the +UK budget, and the original [Where Does My Money Go?](http://wheredoesmymoneygo.org) +page was based on this data. + +The first call we'll make will aggregate the complete dataset +and give us a total sum ([result](http://openspending.org/api/2/aggregate?dataset=ukgov-finances-cra)): + + GET /api/2/aggregate?dataset=ukgov-finances-cra + +This is not very useful, however, as it includes UK spending +over several years. So let's refine our query to include only +2010 figures ([result](http://openspending.org/api/2/aggregate?dataset=ukgov-finances-cra&cut=time.year:2010)): + + GET /api/2/aggregate?dataset=ukgov-finances-cra&cut=time.year:2010 + +Much better! Now we may want to know how these funds are distributed +geographically, so let's drill down by the [NUTS](http://epp.eurostat.ec.europa.eu/portal/page/portal/nuts_nomenclature/introduction) +names of each region of the UK ([result](http://openspending.org/api/2/aggregate?dataset=ukgov-finances-cra&cut=time.year:2010&drilldown=region)): + + GET /api/2/aggregate?dataset=ukgov-finances-cra&cut=time.year:2010&drilldown=region + +Given an SVG file with the right region names, this could easily be +used to drive a CSS-based choropleth map, with a bit of JavaScript +glue on the client side. + +Another set of dimensions of the CRA dataset is the [Classification of +Functions of Government (COFOG)](http://unstats.un.org/unsd/cr/registry/regcst.asp?Cl=4), +which classifies government activity by its functional purpose. Like +many taxonomies, COFOG has several levels, which we have modelled as +three dimensions: cofog1, cofog2 and cofog3. + +In order to generate a [BubbleTree](http://vis4.net/blog/posts/tutorial-bubble-tree/) +diagram, we want to break down the full CRA dataset by each of these +dimensions ([result](http://openspending.org/api/2/aggregate?dataset=ukgov-finances-cra&cut=time.year:2010&drilldown=cofog1|cofog2|cofog3)): + + GET /api/2/aggregate?dataset=ukgov-finances-cra&cut=time.year:2010&drilldown=cofog1|cofog2|cofog3 + +(Warning: this generates quite a lot of data. You may want to paginate +the results to view it in your browser.) + +As you can see, the aggregator API can be used to flexibly query the +data to generate views such as visualizations, maps or pivot tables. + +**Up**: [OpenSpending API](../) diff --git a/examples/openspending/content/help/conventions.md b/examples/openspending/content/help/conventions.md new file mode 100644 index 00000000..5d190049 --- /dev/null +++ b/examples/openspending/content/help/conventions.md @@ -0,0 +1,48 @@ +--- +section: help +lead: true +title: API conventions +authors: +- Neil Ashton +redirect_from: +- /help/api/ +--- +#### Authentication + +Some actions in OpenSpending require authentication, particularly those that write to the system or aim to access protected data (e.g. pre-publication datasets). For this purpose, each user is provided an API key. The key is displayed in the *settings* (go to the dashboard and click on *Change* next to the Information header). You can use it to perform authentication by adding the following into the HTTP headers (change to the API key you find in your settings): + + Authorization: ApiKey + +#### JSON-P Callbacks + +All API calls that return JSON support JSON-P (JSON with padding). You can +add a ``?callback=foo`` parameter to any query to wrap the output in a +function call. This is used to include JSON data in other sites that do not +support CORS: + + $ curl http://openspending.org/cra.json?callback=foo + + foo({ + "description": "Data published by HM Treasury.", + "name": "cra", + "label": "Country Regional Analysis v2009", + "currency": "GBP" + }); + +This can be used in remote web pages to include data as a simple ``script`` +tag: + + + + + + +**Up**: [OpenSpending API](../) diff --git a/examples/openspending/content/help/guide/adding-data-overview/index.md b/examples/openspending/content/help/guide/adding-data-overview/index.md new file mode 100644 index 00000000..e5dee22a --- /dev/null +++ b/examples/openspending/content/help/guide/adding-data-overview/index.md @@ -0,0 +1,28 @@ +--- +section: help +lead: true +title: 'Adding data: overview' +authors: +- Neil Ashton +--- +One of the most valuable contributions you can make to the OpenSpending project is to add a new dataset. This section of the guide walks you through the process of adding new data. + +A typical workflow for importing a dataset into OpenSpending involves the following steps: + +* Gather machine-readable data from a trustworthy source. + +* Convert the data to a CSV file in the format expected by OpenSpending, cleaning it to remove inconsistencies and errors. + +* Publish the data to the web. + +* Create a dataset add the published data as a new data source. + +* Model the dataset to assign a logical role to each column in the source table. + +* Load the data, or refine the data based on the feedback given by the platform about the data's consistency. + +Each of these steps will be explained in detail in the following sections. + +**Next**: [Gathering data](../gathering-data/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/create-viz/index.md b/examples/openspending/content/help/guide/create-viz/index.md new file mode 100644 index 00000000..0ce71336 --- /dev/null +++ b/examples/openspending/content/help/guide/create-viz/index.md @@ -0,0 +1,50 @@ +--- +section: help +lead: true +title: Create a Visualization +authors: +- Neil Ashton +--- +The OpenSpending platform makes it easy to create and embed visualizations of datasets. Three types of visualizations are supported: BubbleTree, TreeMap, and Table of Aggregates. +All OpenSpending visualizations allow you to choose a series of dimensions along which to aggregate your data, drilling down into finer and finer detail. Each visualization is created the same way: by choosing the dimensions to aggregate and the order in which to drill down. +To start creating a visualization, go to a dataset's home page and select **Create a visualization** from the *Visualizations* menu. + +#### BubbleTree + +The BubbleTree is an interactive visualization that presents aggregated spending data as a circle of bubbles. Each bubble represents an aggregated (sub-)total. The central bubble represents an aggregated sum, and its surrounding bubbles represent the other sums that it is made of. By clicking on any bubble, the user is shown how the sum breaks down into further sub-totals. + +To create a BubbleTree, choose the dimensions to aggregate and the order in which to aggregate them. Choose the primary dimension from the *Level* drop-down menu. You will see the aggregated total of that dimension as the central bubble, with values of the dimension surrounding it with their own totals. + +image_14 + +To add a second level, click **Add a level** and choose a new dimension. Users will now be able to click on bubbles to "drill down" and see how the values of the first level break down into values on the second level. + +image_15 + +#### TreeMap + +The TreeMap presents aggregated spending data as an interactive rectangle of coloured tiles. Each tile represents aggregated values for a particular dimension of the data. Clicking on the tile "zooms in" to show how it breaks down along further aggregated dimensions. + +To create a TreeMap, simply choose the dimensions to aggregate and their order. Select the primary dimension from the *Tile* menu. You will see a TreeMap showing how the total spending breaks down across that dimension. + +image_16 + +The visualization has no useful interactivity yet. Adding further tile levels allows you to drill down to see how aggregated values decompose into smaller aggregates. To add a second level of tiles, click **Add a level** and choose a new dimension. Users can now click tiles to see how their totals break down. + +image_17 + +#### Table of Aggregates + +The Table of Aggregates is a simple tabular view of a dataset that aggregates totals across chosen dimensions. A Table of Aggregates is specified by choosing dimensions for its columns. + +Choosing a primary dimension via the *Columns* menu will display the data in tabular form, with aggregated amounts and percentages of the overall total. By default, the rows will be sorted by percentage. + +image_18 + +Adding another column by clicking **Add a level** will break down each subtotal in the first column by the aggregated sums of the new column. Note that this generally changes the percentage values and thus rearranges rows. + +image_19 + +**Next**: [Embed a visualisation into your website](../embed-viz) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/creating-dataset/index.md b/examples/openspending/content/help/guide/creating-dataset/index.md new file mode 100644 index 00000000..1ff6b429 --- /dev/null +++ b/examples/openspending/content/help/guide/creating-dataset/index.md @@ -0,0 +1,54 @@ +--- +section: help +lead: true +title: Creating a dataset on OpenSpending +authors: +- Neil Ashton +--- +To begin sharing data on the OpenSpending platform, register on OpenSpending.org and create a new OpenSpending dataset. To create a dataset, simply fill in some metadata that characterizes your data and provide the URL where your data is hosted. + +#### Creating a new dataset + +Log in to OpenSpending.org with your user information, or register if you have not yet done so. You will arrive at the Dashboard, where you will see a blue button labeled **Import a Dataset**. Click this to begin creating a new OpenSpending dataset. + +The next screen prompts you to provide metadata that characterizes your data. This includes the following fields: + +* *Title*: a descriptive and meaningful name for the dataset. Can be any string. + +* *Identifier*: a shorter title, used as part of the dataset's URL. Can only contain alphanumeric characters, dashes, and underscores – no whitespace or punctuation. + +* *Category*: one of "Budget", "Expenditure", and "Other". See the guide section on types of financial data for details on these categories. + +* *Currency*: the currency in which the spending described by the dataset takes place. + +* *Countries*: a list of countries referenced in the dataset. Choice of countries is constrained by a list of valid countries. + +* *Languages*: a list of languages used in the dataset. Choice of languages is constrained by a list of valid languages. + +* *Description*: a characterization of the dataset in simple prose. Can be any string. + +Fill in all of these fields. Be sure to include a Description which explains the origin of your dataset and acknowledges any changes you have introduced (for example, any cleaning you have done). + +Once all metadata has been filled in, press **Next Step** to proceed. + +#### Adding a new data source + +Clicking through to the next step creates your new OpenSpending dataset and takes you to its *Manage* page. The Manage page is used to add data sources. It is also used to provide schematic information that allows OpenSpending to interpret the data, a process called "modelling" that will be covered in the next section of the guide. + +To add a data source to a dataset, click **Add a source**. A prompt will appear, asking you for a URL. Provide the URL of the CSV file you published on the web in the previous section of the guide and click **Create**. You will see a blue text box indicating that OpenSpending is thinking about your data. + +image_2 + +Click **Refresh** or simply use your browser's refresh button. If OpenSpending succeeded at analyzing your data, you should see a green text box telling you that your data is ready. You should also see a correct list of your CSV's columns. + +image_3 + +Note that if you incorrectly provide OpenSpending with an HTML file instead of a valid CSV file, it will not complain but will simply try to analyze the HTML as if it were a CSV. The result looks like the following. + +image_4 + +If you added a bad data source, don't worry. You do not have to use the source in your final dataset: OpenSpending requires you to do more work on a data source before it can be published. Simply add a new, correct source and forget about the bad one. + +**Next**: [Modelling your data in OpenSpending](../modelling-data/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/data-model/index.md b/examples/openspending/content/help/guide/data-model/index.md new file mode 100644 index 00000000..2db83859 --- /dev/null +++ b/examples/openspending/content/help/guide/data-model/index.md @@ -0,0 +1,28 @@ +--- +section: help +lead: true +title: How does OpenSpending represent data? +authors: +- Neil Ashton +--- +OpenSpending maintains a collection of datasets, each of which represents a set of data from a separate source. Inside each dataset, individual transactions are represented by a set of entries. Each dataset has its own model that maps the structure of the data. The model encodes the properties of each dataset entry in terms of *dimensions*. + +#### Datasets + +The basic unit in the OpenSpending system is the dataset. Financial transactions sharing a common theme (e.g. a particular city’s spending, a budget for a particular year) are grouped together and stored as a dataset. A dataset is a collection of "entries", and each entry represents a single transaction associated with a quantity of money and a time. + +Datasets also include metadata to characterize their contents. The metadata includes a description of the dataset, information about the source of the data, and other such information which helps users find the dataset and interpret its contents. + +#### Models + +The structure of each dataset is completely up to the creator of the dataset. This structure is created by specifying a *model*, which provides the dimensions along which entries can differ from one another. + +A model consists of a set of *dimensions*. A dimension is a property that potentially differentiates one entry from another. If you imagine a dataset as a spreadsheet, each dimension can be thought of as a column. Dimensions can have more structure than an ordinary spreadsheet column, however. + +Dimensions come in several types. The most important is the *measure* type. Measures are dimensions which can contain a single numerical value. Another important dimension type is the *time* type, which represent dates and times. Every data needs at least one each of measure and time dimensions, representing respectively the amount of money represented by the transaction and the time when it took place. + +The remaining dimension types are used to represent other properties that entries might have, e.g. transaction numbers, labels from a classification scheme, or the names of individuals or companies involved. Such dimensions include *attributes*, which can hold a single value, and *compound dimensions*, which can hold a nested set of values. Compound dimensions are useful when a property includes several sub-properties which could each be used to aggregate the data. + +**Next**: [Adding data: overview](../adding-data-overview/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/embed-viz/index.md b/examples/openspending/content/help/guide/embed-viz/index.md new file mode 100644 index 00000000..937183df --- /dev/null +++ b/examples/openspending/content/help/guide/embed-viz/index.md @@ -0,0 +1,16 @@ +--- +section: help +lead: true +title: Embed a visualization into your website +authors: +- Neil Ashton +--- +You can easily embed any of the visualizations created on OpenSpending on your own website. This means you can have the full interactive displays on your site. + +Let's say you have chosen a visualization on the OpenSpending platform. Notice there's an **Embed** button at the bottom right of the webpage. Click this button and you'll be presented with the code to embed the visualization on your website and some options for the size (in pixels) of the interactive. The rest is just cutting and pasting this code into your site. If you are unsure how to paste the code correctly, contact your site administrator. + +The reason it's possible to embed code comes down to *widgets*. In very simplified terms, a widget is a piece of code you can add into your webpage, and it pulls data – in this case, from the OpenSpending database – so you don't need to store datasets yourself. + +**Back to top**: [OpenSpending Guide](../) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/financial-data-types/index.md b/examples/openspending/content/help/guide/financial-data-types/index.md new file mode 100644 index 00000000..3ed7647a --- /dev/null +++ b/examples/openspending/content/help/guide/financial-data-types/index.md @@ -0,0 +1,48 @@ +--- +section: help +lead: true +title: What types of financial data can go into OpenSpending? +authors: +- Neil Ashton +--- +OpenSpending is very flexible in the types of financial data it supports. Although the OpenSpending project has a strong focus on government finance, this is not a technical constraint. OpenSpending supports any dataset consisting of a set of transactions, each associated with a quantity of money and a time. + +Most of the data currently hosted on OpenSpending can be categorized as either transactional or budgetary data. The main difference between these is their level of granularity. Transactional data tracks individual transactions, whereas budgetary data aggregates transactions into categories. + +#### Transactional spending data + +Transactional data, or simply "spending data", tracks individual financial transactions. Each payment from one entity to another on a given date and for a specific purpose (e.g. a project or service) is listed individually. Transactional spending data includes various types of records, including information on government grants, commitments, and actual expenditure. + +Aggregate information (e.g. subtotals) should not be included in transactional data. Data that has been partially or completely aggregated calls for a different mode of analysis and should be treated as budgetary data rather than transactional data. This does not mean, however, that several "physical" payments which amount to a single “logical” transaction cannot be represented by a single transaction in transactional data. + +Transactional data on OpenSpending includes: + +* [DC Vendors and Contractors](http://openspending.org/dc-vendors-contractors) + +* [Austrian Development Agency](http://openspending.org/ada/) + +Another related type of data deals with the public procurement procedures. Public Procurement data is data about public tenders: what was tendered, for how much, and who won the tender. It can be seen as a subset of transactional data. + +Procurement data on OpenSpending includes: + +* [Marchés publics au Sénégal](http://openspending.org/marches-publics-senegal/views/liste-des-attributaires) + +* [Marchés publics France 2011](http://openspending.org/marches-publics-france-2011) + +#### Budgetary data + +In budgetary data, expenditures and incomes are aggregated into categories. The goal of this aggregation is to aid the reader in understanding the budget, which is typically a policy document that is used to provide readers with an overview of the government’s most important financial choices. Allocation is typically structured by a classification scheme rather than by the actual recipients of funds. + +Budgetary data often jointly presents data on past outcomes and appropriations for a future period. In such a presentation, amounts spent in previous years on a particular sector are used to inform how much should be allocated for the coming budgeting period. Budget information is often based on aggregated data and statistical estimates. + +Different regions make different types of budgetary information available, including: Pre-Budget Statements; Executive Budget Proposals; Enacted Budgets; and Citizen's Budgets (simplified versions of the budget for the benefit of citizens). + +Budgetary data on OpenSpending includes: + +* [Berlin Budget](http://openspending.org/berlin_de) + +* [Seville Spending Budget](http://openspending.org/seville-budget) + +**Next**: [How does OpenSpending represent data?](../data-model/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/formatting-data/index.md b/examples/openspending/content/help/guide/formatting-data/index.md new file mode 100644 index 00000000..21d7a6c3 --- /dev/null +++ b/examples/openspending/content/help/guide/formatting-data/index.md @@ -0,0 +1,52 @@ +--- +section: help +lead: true +title: Formatting data +authors: +- Neil Ashton +--- +OpenSpending expects all data to be in a simple format. + +#### CSV + +OpenSpending accepts data in a single file format, the Comma-Separated Values (CSV) file. A CSV is a plain text file that represents data as a table, which is similar to a spreadsheet. In a table, each data point is represented by a row, and each data point's properties are represented by a column. CSV files encode tables by giving each row a line in the text file and by separating columns with commas. + +CSVs accepted by OpenSpending do not save space by removing redundant values. If your spreadsheet omits any repeated values, those omitted values must be filled in before OpenSpending can use your data. OpenSpending-ready CSVs are also *rectangular*, meaning that they have exactly the same number of columns in each row. + +#### The OpenSpending format + +CSVs for OpenSpending must have the following properties. + +1. One header row. The first row of the CSV file should contain the names of the columns, separated by commas. All other rows are treated as data rows. + +2. At least three columns. The bare minimum of columns are an amount, a date (which could be just a year), and a spender or a recipient (which could just be the name of an account). + +3. Consistent columns. Each column must consistently represent a single type of value for all rows. (There can be no subheader rows, for example.) + +4. Rows are single data points. Rows should contain *only one* transaction or one budget line. Each row must represent a maximum of one time period. + +5. No blank rows or cells. Each row should be completely filled in. Some spreadsheets leave redundant data cells blank or have other ways of saving space, but OpenSpending requires each row to be complete on its own. + +6. No pre-aggregated totals (e.g. sub-totals or "roll-ups"). OpenSpending will do the maths and compute these automatically. + +7. Rows have values that uniquely identify them. Each row must have some column (or combination of column) whose value(s) can be used as an "ID" for the row. Each row's ID must be unique. For example, your data could have a column named "ID" which contains a different number for each row. An easy way to create such an ID column in Excel is to add a new column, write "1" in the top cell of the column, write "2" in the second cell from the top, select both cells, and then click and drag the lower right corner of the selection to the bottom of the spreadsheet. + +8. Dates in the correct format. Dates must be in the format "yyyy-mm-dd". + +9. Numbers in the correct format. Numbers must contain only digits and an optional period—no commas! (Readable numbers like "12,345.67" should be converted to numbers like “12345.67”.) + +The OpenSpending community has gathered some [example spreadsheets](https://drive.google.com/a/okfn.org/#folders/0B_dkMlz2NopEbmRoTExsMDFMR2M) in order to illustrate what "good" and “bad” tabular data looks like. Here are some examples of badly formatted spreadsheets: + +* [Many blank cells](https://docs.google.com/a/okfn.org/spreadsheet/ccc?key=0AvdkMlz2NopEdEtIMFlEVDZXOWdDUEthUTQ0c21aV2c#gid=0) (probably redundant info omitted) + +* [Multiple transactions, one row](https://docs.google.com/a/okfn.org/spreadsheet/ccc?key=0AvdkMlz2NopEdG5kR0kzQ0E5V3BuTS16MndBT3dMdEE#gid=0) (multiple years on one row) + +* [Bad numbers](https://docs.google.com/a/okfn.org/spreadsheet/ccc?key=0AvdkMlz2NopEdEo1Y2p2R0VvdnJvRXMwUVREbHRoLXc#gid=0) (numbers have commas for readability) + +Here is a good spreadsheet: + +* [Washington, DC](https://docs.google.com/a/okfn.org/spreadsheet/ccc?key=0AvdkMlz2NopEdDhrZnRkWl9ZX2ZZNVptTzdueWw3emc#gid=0) + +**Next**: [Publishing data on the web](../publishing-data) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/gathering-data/index.md b/examples/openspending/content/help/guide/gathering-data/index.md new file mode 100644 index 00000000..459e123d --- /dev/null +++ b/examples/openspending/content/help/guide/gathering-data/index.md @@ -0,0 +1,16 @@ +--- +section: help +lead: true +title: Gathering data +authors: +- Neil Ashton +--- +To add a dataset to OpenSpending, you must first have some data. If you already have it, you can proceed. If not, you need to find it. + +Begin your search for data by consulting esources such as the[ School of Data](http://schoolofdata.org/handbook/courses/finding-data/) and the[ Data Journalism Handbook](http://datajournalismhandbook.org/1.0/en/getting_data.html). You can also get ideas on how to go about your search by visiting the[ OpenSpending group](http://datahub.io/group/openspending) on datahub.io, and you can ask questions on the #openspending IRC channel on Freenode. + +The data you find will hopefully be in a "machine-readable" format, for example in the form of an Excel spreadsheet or a CSV file. If you find data in a format like PDF or a Word document, it will be very hard to work with, and you might consider simply trying different data! + +**Next**: [Formatting data](../formatting-data/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/index.md b/examples/openspending/content/help/guide/index.md new file mode 100644 index 00000000..abf1ee32 --- /dev/null +++ b/examples/openspending/content/help/guide/index.md @@ -0,0 +1,27 @@ +--- +section: help +lead: true +title: The OpenSpending Guide +authors: +- Neil Ashton +--- +The OpenSpending Guide is the manual for OpenSpending, covering the entire process of finding, adding, and using data with OpenSpending in detail. + +* Introduction + * [What is OpenSpending?](./what-is-openspending) + * [What types of financial data can go into OpenSpending?](./financial-data-types) + * [How does OpenSpending represent data?](./data-model) +* Adding Data to OpenSpending + * [Overview](./adding-data-overview) + * [Gathering data](./gathering-data) + * [Formatting data](./formatting-data) + * [Publishing data on the web](./publishing-data) + * [Creating a dataset on OpenSpending](./creating-dataset) + * [Modelling your data in OpenSpending](./modelling-data) +* Visualizations + * [Create a visualization](./create-viz) + * [Embed a visualization on your website](./embed-viz) + +**Begin**: [What is OpenSpending?](./what-is-openspending/) + +**Up**: [OpenSpending guides](../) diff --git a/examples/openspending/content/help/guide/modelling-data/index.md b/examples/openspending/content/help/guide/modelling-data/index.md new file mode 100644 index 00000000..a17498b0 --- /dev/null +++ b/examples/openspending/content/help/guide/modelling-data/index.md @@ -0,0 +1,76 @@ +--- +section: help +lead: true +title: Modelling your data in OpenSpending +authors: +- Neil Ashton +--- +To load data into OpenSpending, you must build a *model* of your data. A model specifies how your data translates into terms OpenSpending understands. OpenSpending represents the properties data in terms of *dimensions*. Modelling data consists of listing the dimensions you would like the target OpenSpending dataset to have and specifying how they relate to columns in the source data. + +#### Mandatory dimensions: amount and time + +Every model needs to have at least two dimensions: an amount and a time. These specify the size of the transaction and the time when the transaction took place. The amount and time are associated with special types of dimensions. An amount is represented by a *measure* dimension, and a time is represented by a *date*. Generic dimensions cannot represent these special values. + +When modelling your data, it's not a bad idea to start with the mandatory dimensions. To begin, click the **Dimensions & Measures** tab within your dataset's **Manage the dataset** page. + +image_5 + +Next, click **Add Dimension** to bring up the *Add new dimension* panel. Click the radio button labeled *Date*. You will see the *Name* box automatically fill with "time", as shown below. Click the green **Add** button. + +image_6 + +The next screen you see will provide you some information about the meaning of time. In the drop-down box next to *Column:*, select the column of your data which represents the time value. + +image_7 + +Once you identify the time column, click **Add Dimension** once again to add the amount. This time, select the radio button labeled *Measure*, which will automatically fill in the name "amount", and click **Add**. Choose the column representing the value of the transaction from the drop-down box next to *Column*. + +#### The key and compound dimensions + +Only one additional dimension is necessary to make the model sufficient: the dimension (or set of dimensions) whose value uniquely identifies each data point, the *key*. + +A data point does not need to be identified by the value of a single column. It can be identified by the combination of several in a *compound dimension*. Because keys *can* be compound, the compound dimension type *must* be used to represent them, even if your particular key is not compound. + +To add the key dimension, click **Add Dimension** and select the *Dimension* radio button. Enter a name for your key, such as "key", in the name box. Click **Add**. Check the box labeled *Include in unique key* to identify this dimension as part of your key. + +Next, take a look at the list of **Fields**, which contains two rows labeled *name *and *label*. A compound dimension can contain an arbitrary number of *fields*, each of which has a name and a type and each of which can be associated with a column in your data. This is the sense in which these dimensions are "compound": they group multiple columns from the source data into a single property of the target dataset. + +image_8 + +A compound dimension requires at least two fields, *name* and *label*, which must respectively be of type *id* and *string*. The dimension's name is used to provide it with a working URL, and the label is used to present it in the user interface. + +To create a minimal compound dimension, simply associate the same column of the source data with both *name* and *label*. Choose the appropriate column for each and leave the default types unchanged. + +#### Measures and other dimensions + +With an amount, time, and key, your model is sufficiently rich. A really complete model, however, will include dimensions for every meaningful property of the source data. Following certain conventions makes this more convenient. + +A common pattern in source data is spreading information that identifies entities – groups, accounts, and so on – across multiple columns. Information about an account associated with a transaction may be divided into an "Account" column with an identifying number and an "Account description" column with a verbal description, for example. "Head-account" and "Sub-account" in the image below exhibit this pattern. + +image_9 + +OpenSpending's compound dimensions are designed to model this kind of scattered information. To do so, add a new compound dimension and associate each column to one of the dimension's fields. Try to match a human-readable column to *label* and a more terse column to *name*. In the image below, "Head-account" is matched to *name* and "Head-account description" to *label*. + +image_10 + +Some columns of your data are more self-contained, representing particular attributes of each data point. A column which sorts each transaction into some category, for example, is of this type. In the image below, the Reporting Type, Revenue/Expenditure, and Recurrent/Investment columns are like this. + +image_11 + +Self-contained columns specifying attributes or categories are best modeled with *attribute* dimensions. An attribute is essentially a dimension with only a single field, which may have any type. To create an attribute, simply select the *Attribute* radio button when adding a dimension. + +image_12 + +#### Wrapping up: saving and loading + +When every dimension has been specified and linked to columns in the source data, click **Save Dimensions** to save the model. If anything is wrong with the model, an error message will appear, prompting you to correct its parameters. Otherwise, a message will appear inviting you to return to the dashboard, where you can proceed to load your data. + +Once the data has been loaded, the model you have created will be fixed and editing will be disabled. You may therefore wish to test the model before you load. To do this, click **Test a sample** in your data source's row in the dashboard. Wait a few seconds, then reload the page. If you see a message saying COMPLETE with a green background, your model is ready to go. If you see ERRORS, repairs are needed. + +image_13 + +If your model is free of errors, click **Load** to load the source dataset and apply the model. You may then return to the dataset's home page by clicking its name at the top of the screen, where you can proceed to construct visualizations and otherwise play with your data. + +**Next**: [Create a Visualization](../create-viz/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/publishing-data/index.md b/examples/openspending/content/help/guide/publishing-data/index.md new file mode 100644 index 00000000..1c54ba8f --- /dev/null +++ b/examples/openspending/content/help/guide/publishing-data/index.md @@ -0,0 +1,38 @@ +--- +section: help +lead: true +title: Publishing data on the web +authors: +- Neil Ashton +--- +Data cannot (yet) be uploaded directly to OpenSpending. In order to be added to the OpenSpending database, data must first be made accessible from the web. This section introduces two convenient ways to publish sets of data online. + +#### Google Drive + +You can make your data accessible on the web by turning it into a Google Drive spreadsheet. + +1. Import your data. Create a new Google Drive spreadsheet, then select *Import...* from the File menu. Select *Replace Spreadsheet*, click **Choose File**, and navigate to your CSV file. + +2. Make sure Google Docs doesn't mis-format your data's dates. Select the column that contains dates. Click the *Format* menu and select *Number* -> *More formats* -> *2008-09-26*. Your dates should appear in the prescribed **yyyy-mm-dd** format. + +3. Click the *File* menu and select *Publish to the web...*. In the box that appears, click **Start publishing**. Beneath *Get a link to the published data*, select **CSV (comma-separated values)**. + +image_0 + +The URL at the bottom of the box now points to your data. + +#### Gist + +GitHub Gist is a convenient way to host small quantities of text, including CSV files. + +1. Log in to GitHub (or register if you haven't already done so), then navigate to[ gist.github.com](https://gist.github.com/). + +2. Click and drag your CSV file from your operating system's file manager onto the GitHub Gist page of your browser. The file's name and contents will appear. + +3. Click **Create Public Gist** to be taken to the homepage of your new gist. The raw URL of your data is accessible through the "angle brackets" button in the top right corner of the file. + +image_1 + +**Next**: [Creating a dataset on OpenSpending](../creating-dataset/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/guide/what-is-openspending/index.md b/examples/openspending/content/help/guide/what-is-openspending/index.md new file mode 100644 index 00000000..1d867d77 --- /dev/null +++ b/examples/openspending/content/help/guide/what-is-openspending/index.md @@ -0,0 +1,14 @@ +--- +section: help +lead: true +title: What is OpenSpending? +authors: +- Neil Ashton +--- +OpenSpending is a data sharing community and web application that aims to track every government and corporate financial transaction across the world and to present that data in a useful and engaging form. OpenSpending is an open project maintained by a community of contributors. Anyone interested in spending data of any kind is invited to contribute data to the OpenSpending database, create visualizations using the OpenSpending software, and use the OpenSpending API. + +This chapter introduces new OpenSpending contributors to the core concepts of the system. It describes the kind of financial data that OpenSpending supports, and it explains how OpenSpending represents that data. + +**Next**: [What types of financial data can go into OpenSpending?](../financial-data-types/) + +**Up**: [OpenSpending Guide](../) diff --git a/examples/openspending/content/help/images/browse_teaser.png b/examples/openspending/content/help/images/browse_teaser.png new file mode 100644 index 00000000..954b03b9 Binary files /dev/null and b/examples/openspending/content/help/images/browse_teaser.png differ diff --git a/examples/openspending/content/help/images/controls.png b/examples/openspending/content/help/images/controls.png new file mode 100644 index 00000000..3098900e Binary files /dev/null and b/examples/openspending/content/help/images/controls.png differ diff --git a/examples/openspending/content/help/images/engine.png b/examples/openspending/content/help/images/engine.png new file mode 100644 index 00000000..1913f17a Binary files /dev/null and b/examples/openspending/content/help/images/engine.png differ diff --git a/examples/openspending/content/help/images/kit_teaser.png b/examples/openspending/content/help/images/kit_teaser.png new file mode 100644 index 00000000..bcd34fce Binary files /dev/null and b/examples/openspending/content/help/images/kit_teaser.png differ diff --git a/examples/openspending/content/help/images/mapping.png b/examples/openspending/content/help/images/mapping.png new file mode 100644 index 00000000..2d70a5d6 Binary files /dev/null and b/examples/openspending/content/help/images/mapping.png differ diff --git a/examples/openspending/content/help/images/upload_teaser.png b/examples/openspending/content/help/images/upload_teaser.png new file mode 100644 index 00000000..2d8c33cb Binary files /dev/null and b/examples/openspending/content/help/images/upload_teaser.png differ diff --git a/examples/openspending/content/help/index.md b/examples/openspending/content/help/index.md new file mode 100644 index 00000000..e712d3a2 --- /dev/null +++ b/examples/openspending/content/help/index.md @@ -0,0 +1,49 @@ +--- +title: Help +authors: +- Anders Pedersen +redirect_to: "/get-involved/" +--- + +Are you looking to get started using OpenSpending? Have a look at some +of our guides and get rolling! If you have questions, +[get in touch](../about/contact). + +The OpenSpending Guide is available in multiple languages: + +* [English](./guide/en) +* [French](./guide/fr) +* [Italian](./guide/it) +* [Nepali](./guide/npl) +* [Romanian](./guide/rom) +* [Spanish](./guide/esp) + +Many other translations are under development. Check back for new +translations or +[help add your language](https://trello.com/c/abfAVgBC/14-permanent-openspending-guide-translation). + +#### Video guides + +* [OpenSpending Upload, Step 1](http://vimeo.com/43259079#): adding a dataset +* [OpenSpending Upload, Step 2](http://vimeo.com/43760979): data modelling +* [How to Fix Errors in OpenSpending](http://vimeo.com/43762097) +* [Introduction to Spending Stories](http://webtv.journalismfestival.com/doc/1311/raccontare-la-spesa-pubblica.htm) +* [Talk from "The Big Clean", Prague 2012](http://www.youtube.com/watch?v=ofzU43g5ZpY) + +#### Tutorials + +* [D3.js Sankey diagrams with the OpenSpending API](http://blog.openspending.org/2013/08/28/d3-sankey/) + +#### OpenSpending API + +OpenSpending aims to be a simple-to-use platform for developers by +offering a variety of APIs that can be used to create visualizations, +reports, and other applications. These help pages describe the APIs +and provide examples of their use. + +* [API conventions](./conventions) +* [Aggregate API](./aggregate) +* [REST resources](./rest) +* [Full-text search API](./search) +* [Personal tax API](./tax) +* [Dataset loading API](./loading) diff --git a/examples/openspending/content/help/links.md b/examples/openspending/content/help/links.md new file mode 100644 index 00000000..65c2bedf --- /dev/null +++ b/examples/openspending/content/help/links.md @@ -0,0 +1,25 @@ +--- +section: help +lead: true +title: Key Community Resources +authors: +- Neil Ashton +--- + +OpenSpending Community key shared resources. + +#### Documents + +* [Community Google Drive folder](https://drive.google.com/a/okfn.org/#folders/0B6R8dXc6Ji4JUWs5UDdzSzU2UG8) + +#### Data Stuff + +* [Tracker for data to load](https://github.com/openspending/datatoload) + +#### Code + +[OpenSpending on GitHub](https://github.com/openspending) + +* [Core application](https://github.com/openspending/openspending) +* [JavaScript library](https://github.com/openspending/openspendingjs) +* [Satellite site template](https://github.com/openspending/satellite-template) diff --git a/examples/openspending/content/help/loading.md b/examples/openspending/content/help/loading.md new file mode 100644 index 00000000..92448e81 --- /dev/null +++ b/examples/openspending/content/help/loading.md @@ -0,0 +1,18 @@ +--- +section: help +lead: true +title: Dataset loading API +authors: +- Neil Ashton +--- + +#### OpenSpending Loading API + +Users can load datasets (or add sources to them) by making a *POST* request to ``https://openspending.org/api/2/new`` (notice *https*) with the following url parameters: + +* *csv_file* - A **url** to the csv file to me imported for the dataset +* *metadata* - A **url** to the json file with dataset metadata (name, currency, etc.) and the model. Views can also be defined in this file. Take a look at [a sample json file](https://dl.dropbox.com/u/3250791/sample-openspending-model.json) to see how it should be structured (the value for *mapping* is the model - how the csv file should be cast into dataset dimensions, and the value for *dataset* is the metadata itself). To gain a better understanding of how to do the mapping, take a look at [the corresponding csv file](http://mk.ucant.org/info/data/sample-openspending-dataset.csv). + +Along with these two an api key must be provided in the header of the request. For more details see [API Conventions](/help/conventions/). + +**Up**: [OpenSpending API](../) diff --git a/examples/openspending/content/help/permissions.md b/examples/openspending/content/help/permissions.md new file mode 100644 index 00000000..431f0569 --- /dev/null +++ b/examples/openspending/content/help/permissions.md @@ -0,0 +1,23 @@ +--- +section: help +lead: true +title: Permissions API +authors: +- Tryggvi Björgvinsson +--- +OpenSpending allows users to check for their permissions on a given dataset via an API call. The response will provide the authenticated user's permission on as true or false values for *CRUD* (create, read, update, and delete). This API call mainly exists to allow software that uses the API (e.g. [the loading API](/help/api/loading)) to save bandwidth with big dataset updates. + +For example if you as a developer are building a loading script that users of OpenSpending can use to download data from a location and update datasets in OpenSpending you might first run a check for permissions based on their [API key](http://community.openspending.org/help/api/conventions) before starting to download the updates (so you can skip it if they're not authorized. + +The permission API works as follows. Make a *GET* request (wih user authenticated with the API key) to: + + /api/2/permissions?dataset=[dataset_name] + +The response will be single json object with four properties, *create*, *read*, *update*, and *delete*. The value of each property is a boolean (true or false) that indicates if the authenticated user has that permission for the provided dataset: + + { + "create": false, + "read": true, + "update": false, + "delete": false + } diff --git a/examples/openspending/content/help/rest.md b/examples/openspending/content/help/rest.md new file mode 100644 index 00000000..4773a389 --- /dev/null +++ b/examples/openspending/content/help/rest.md @@ -0,0 +1,197 @@ +--- +section: help +lead: true +title: REST resources +authors: +- Neil Ashton +--- +OpenSpending pages generally support multiple representations, at least +a user-facing HTML version and a JSON object that represents the contained +data. For various technical and non-technical reasons, most of the data is +read-only. + +Content negotiation can be performed either via HTTP ``Accept`` headers or +via suffixes in the resource URL. The following types are generally +recognized: + +* **HTML** (Hyptertext Markup), MIME type ``text/html`` or any value not + otherwise in use, suffix ``.html``. This is the default representation. +* **JSON** (JavaScript Object Notation), MIME type ``application/json`` and + suffix ``.json``. +* **CSV** (Comma-Separated Values), MIME type ``text/csv`` and suffix + ``.csv``. CSV is only supported where listings can be exported with some + application-level meaning. + +The key resources in OpenSpending are datasets, entries, dimensions, and +dimension members. Each of these has a listing and an entity view that can +be accessed. + +#### Listing datasets + + GET /datasets.json + +All datasets are listed, including their core metadata. Additionally, certain +parameters are given as facets (i.e. territories and languages of the +datasets). Both ``territories`` and ``languages`` can also be passed in as +query parameters to filter the result set. Supported formats are HTML, CSV and JSON. + + "territories": [ + /* ... */ + { + "count": 2, + "url": "/datasets?territories=BH", + "code": "BH", + "label": "Bahrain" + }, + /* ... */ + ], + "languages": /* Like territories. */ + "datasets": [ + { + "name": "cra", + "label": "Country Regional Analysis v2009", + "description": "The Country Regional Analysis published by HM Treasury.", + "currency": "GBP" + }, + /* ... */ + ] + +#### Getting dataset metadata + + GET /{dataset}.json + +Core dataset metadata is returned. This call does not have any +parameters. Supported formats are HTML and JSON. + + { + "name": "cra", + "label": "Country Regional Analysis v2009", + "description": "The Country Regional Analysis published by HM Treasury.", + "currency": "GBP" + } + +Another call is available to get the full model description of +the dataset in question, which includes the core metadata and also +a full description of all dimensions, measures, and views. The +format for this is always JSON:: + + GET /{dataset}/model.json + +#### Listing dataset dimensions + + GET /{dataset}/dimensions.json + +A listing of dimensions, including type, description, and attribute +definitions is returned. This call does not have any parameters. +Supported formats are HTML and JSON. + + [ + { + "name": "from", + "html_url": "http://openspending.org/ukgov-finances-cra/from", + "label": "Paid from", + "key": "from", + "attributes": { + "gov_department": { + "column": null, + "facet": false, + "constant": "true", + "datatype": "constant", + "end_column": null + }, + "name": { + "column": "dept_code", + "facet": false, + "constant": null, + "datatype": "string", + "end_column": null + }, + "label": { + "column": "dept_name", + "facet": false, + "constant": null, + "datatype": "string", + "end_column": null + } + }, + "type": "compound", + "description": "The entity that the money was paid from" + }, + /* ... */ + ] + +#### Listing dimension members + + GET /{dataset}/{dimension}.json + +The returned JSON representation contains the dimension metadata, +including type, label, description and attribute definitions. + + { + "name": "from", + "html_url": "http://openspending.org/ukgov-finances-cra/from", + "label": "Paid from", + "key": "from", + "attributes": { + "gov_department": { + "column": null, + "facet": false, + "constant": "true", + "datatype": "constant", + "end_column": null + }, + "name": { + "column": "dept_code", + "facet": false, + "constant": null, + "datatype": "string", + "end_column": null + }, + "label": { + "column": "dept_name", + "facet": false, + "constant": null, + "datatype": "string", + "end_column": null + } + }, + "type": "compound", + "description": "The entity that the money was paid from" + } + +This call's return includes dimension metadata, but it may be too expensive +to call for just this aspect. + +#### Getting dimension members + + GET /{dataset}/{dimension}/{name}.json + +This will return the data stored on a given member ``name`` of the +``dimension``, including its ``name``, ``label``, and any other +defined attributes. + + { + "id": 2, + "name": "10", + "label": "Social protection", + "description": "Government outlays on social protection ...", + "level": "1" + } + +#### Listing entries in a dataset + +Listing all the entries in a dataset (and offering export functionality) +is handled by the full-text search. See [the search API](../search). + +### Getting an entry + + GET /{dataset}/entries/{id}.json + +This will return a full representation of this entry, including all +measures and all attributes of all dimensions. The entry ``id`` is a +semi-natural key derived from dataset metadata which should be stable +across several loads. + +A CSV representation is available but will only have one row. + +**Up**: [OpenSpending API](../) diff --git a/examples/openspending/content/help/search.md b/examples/openspending/content/help/search.md new file mode 100644 index 00000000..f14265f6 --- /dev/null +++ b/examples/openspending/content/help/search.md @@ -0,0 +1,93 @@ +--- +section: help +lead: true +title: Full-text search API +authors: +- Neil Ashton +--- +OpenSpending supports full-text search as a research tool for +everyone who wants to investigate the spending information kept +in our database. + +It is important to note, however, that search is always performed +on individual entries. More abstract concepts (e.g. "all +health spending in a country over a given year") would mostly be the +result of adding up many individual entries. If your use case +requires that you access such concepts, you may want to look at +the [aggregation API](../aggregate) instead. + +#### Basic call and parameters + + GET /api/2/search?q= + +Calls will return a set of fully JSON serialized entries, query +statistics, and, depending on the other parameters, other data such as +facets. + +The following parameters are recognized: + +* ``q`` + Query string. Will usually search a composite text field but can + be limited to a specific field (i.e. a dimension, attribute, or measure) + with ``field:value``. Boolean operators such as OR, AND, and ±term can also be used. + +* ``dataset`` + Specifies a dataset name to search in. While searching across multiple + datasets is supported, this parameter can be used to limit the scope and + increase performance. It can be used multiple times or multiple + dataset names can be separated with pipe symbols. + +* ``category`` + The dataset category can be used to filter datasets by their type, + e.g. limiting the output to only transactional expenditure (and + excluding any budget items). Valid values include ``budget``, + ``spending``, and ``other``. + +* ``stats`` + Includes solr statistics on measures, namely the average, mean, and + standard deviations. This is generated through the indexed data and + can differ marginally from the + results of the aggregator due to floating point inaccuracies. + Note that aggregations + across datasets with different currencies (or even the same currency + across different years) are possible but must be avoided. + +* ``filter`` + Apply a simple filter of the format ``field:value``. Multiple filters + can be joined through pipes, e.g. ``fieldA:value|fieldB:value``. + +* ``page`` + Page number for paginated results. Defaults to ``1``. + +* ``pagesize`` + Size of a page for paginated results. Defaults to ``10000``. + +* ``facet_field`` + A field to facet the search by, i.e. give all the distinct values of + the field in the result set with the count of how often each occurred. + +* ``facet_page``, ``facet_pagesize`` + Works analogously to the ``page`` and ``pagesize`` parameters but applies + to facets instead. + +* ``expand_facet_dimensions`` + When a compound dimension name is used for a facet, this will return a + full representation of this dimension value for each value. + +If an error is detected, the system will return a simple JSON response +with a list of ``errors`` describing the fault. + +### Solr query syntax + +OpenSpending uses Apache Solr for full-text indexing. Some search +parameters are passed directly to Solr: + + GET /api/2/search?q=money%20measure:[min%20TO%20max]&fq=dimension:value + +Some useful resources to explore the query language of Solr include: + +* [Solr Common Query Parameters](http://wiki.apache.org/solr/CommonQueryParameters) +* [Lucene Query Parser Syntax](http://lucene.apache.org/java/3_4_0/queryparsersyntax.html) +* [Solr Query Syntax](http://wiki.apache.org/solr/SolrQuerySyntax) (Advanced) + +**Up**: [OpenSpending API](../) diff --git a/examples/openspending/content/help/tax.md b/examples/openspending/content/help/tax.md new file mode 100644 index 00000000..bb9b2324 --- /dev/null +++ b/examples/openspending/content/help/tax.md @@ -0,0 +1,50 @@ +--- +section: help +lead: true +title: Personal tax API +authors: +- Neil Ashton +--- +The tax share API estimates a household's tax contribution based on simple +proxy data. The estimate allows for both direct tax (including income tax, +national insurance and council tax) and indirect tax (including VAT, alcohol +and tobacco duty, and fuel duty). + + GET http://openspending.org/api/mytax?income=N + +The basic call accepts a variety of parameters, most of which are optional: + +* ``income`` (required) + Total household income, including all pension and benefits. This is + used to estimate total tax paid, including both direct and indirect + taxation. + +* ``spending`` + Total spending on consumption. + +* ``smoker`` + yes/no + +* ``drinker`` + yes/no + +* ``driver`` + yes/no + +This will generate a simple JSON response of the following form: + + { + "alcohol_tax": 153.04239230064161, + "explanation": [ + "This household income falls between national average income decile 1 (which has average gross household income of 9219.00, and pays 1172.00 in direct tax, 1016.00 in VAT, 1101.00 in smoking taxes, 288.00 in alcohol-related taxes, 150.00 in car-related taxes, and 349.00 in other indirect taxes), and decile 2 (which has average gross household income of 13583.00, and pays 1368.00 in direct tax, 969.00 in VAT, 1085.00 in smoking taxes, 310.00 in alcohol-related taxes, 167.00 in car-related taxes, and 289.00 in other indirect taxes).", + "Therefore, a household with an income of 10000.00 pays approximately 1207.08 in direct tax and 2888.97 in total indirect tax." + ], + "tax": 4096.0439963336394, + "tobacco_tax": 291.93721356553618, + "car_related_tax": 338.26214482126488, + "total_direct_tax": 1207.076993583868, + "vat": 1098.1365719523374, + "total_indirect_tax": 2888.9670027497709 + } + +**Up**: [OpenSpending API](../) diff --git a/examples/openspending/content/help/taxman.mdown b/examples/openspending/content/help/taxman.mdown new file mode 100644 index 00000000..d8fa73cd --- /dev/null +++ b/examples/openspending/content/help/taxman.mdown @@ -0,0 +1,35 @@ +# The Where Does My Money Go - Daily Bread, how we calculate the numbers + +We are often asked, how we calculate the numbers in Daily Bread. + +Information on the sources of data which we use for the Daily Bread UK tax calculator can be found on the Where Does my Money Go Site: + + + +It is possible to build your own version of this for your country - here's what you would need to do so... + + + +# Build your own tax calculator - the recipe + +## Raw ingredients + +* **Information on the budget with functional classifications** (in order to be digestible to the average reader, we suggest that you don't have more than 10 top level items or there will probably be too much information for the user to take in). +* Information on the tax rates and how they are calculated +* Information on any overriding concerns e.g. if you are a smoker or a driver, is your tax rate drastically affected and are you interested in reflecting this in your tax calculator? + +### If you want your tax calculator to resemble Where Does My Money Go? You will also need... + +* Images to reflect the functional areas of the budget in your country N.B. If your budget uses the UN's [COFOG](http://en.wikipedia.org/wiki/Classification_of_the_Functions_of_Government) classifications then it will align with the UK's budget and you will be able to use the images from Where Does My Money Go? These images live in the [Github repo](https://github.com/openspending/wheredoesmymoneygo.org/tree/master/icons). + +## Instructions + +*You'll need a developer with some knowledge of JavaScript (node.js) and someone with a good understanding of the tax system of your country*. + +Our tax calculator runs off an API which we affectionately call ['taxman'](https://github.com/openspending/taxman), which can be adapted to accommodate different tax rates, output types (e.g. individual daily, monthly or yearly tax contributions) and other factors such as indirect taxes, which you may also want to reflect in your calculator. + +## Useful links + +* [Where Does My Money Go? - The Daily Bread](http://wheredoesmymoneygo.org/dailybread.html) +* [Taxman API](https://github.com/openspending/taxman) +* The Taxman API lives online on [OpenSpending](The TaxMan API lives at: http://taxman.openspending.org/) \ No newline at end of file diff --git a/examples/openspending/content/resources/eu/acronyms.md b/examples/openspending/content/resources/eu/acronyms.md new file mode 100644 index 00000000..f830acbe --- /dev/null +++ b/examples/openspending/content/resources/eu/acronyms.md @@ -0,0 +1,40 @@ +--- +title: List of Acronyms +--- + +| **ACP** | African, Caribbean and Pacific (countries) | +| **CAP** | Common Agricultural Policy | +| **CF** | Cohesion Fund | +| **CFP** | Common Fisheries Policy | +| **CSO** | Civil Society Organisation | +| **DG** | Directorate-General | +| **EAFRD** | European Agricultural Fund for Rural Development | +| **EAGF** | European Agricultural Guarantee Fund | +| **EDF** | European Development Fund | +| **ECA** | European Court of Auditors | +| **EFF** | European Fisheries Fund | +| **EMFF** | European Maritime and Fisheries Fund | +| **ERCAS** | European Research Centre for Anti-Corruption and State-Building | +| **ERDF** | European Regional Development Fund | +| **ESIF** | European Structural and Investment Funds | +| **EU** | European Union | +| **FIFG** | Financial Instrument for Fisheries Guidance | +| **FTS** | Financial Transparency System | +| **GNI** | Gross National Income | +| **ICIJ** | International Consortium of Investigative Journalists | +| **IMF** | International Monetary Fund | +| **IMP** | Integrated Maritime Policy | +| **MFF** | Multiannual Financial Framework | +| **OCTs** | Overseas Countries and Territories | +| **OLAF** | European Anti-Fraud Office | +| **TBIJ** | The Bureau of Investigative Journalism | +| **TED** | Tenders Electronic Daily | +| **TFUE** | Treaty on the Functioning of the European Union | +| **TI-EU** | Transparency International EU Office | +| **UN** | United Nations | +| **VAT** | Value-added Tax | + +- [Return to Beginning](../) +- [Next: Glossary](../glossary/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/budget.md b/examples/openspending/content/resources/eu/budget.md new file mode 100644 index 00000000..92a8d7a0 --- /dev/null +++ b/examples/openspending/content/resources/eu/budget.md @@ -0,0 +1,77 @@ +--- +title: How is the EU Budget Set Up? +--- + +Unlike most national budgets which are only set up on an annual basis, the EU budget is set up through two main instruments: the Multiannual Financial Framework (MFF) for the long term, and a more standard annual budget regulation. + +The Multiannual Financial Framework +----------------------------------- + +The MFF is an instrument which sets the maximum annual amounts - or “ceilings” - which the EU may spend in different political fields - or “headings” - for a period of at least 5 years. In practice, the MFF is adopted for a period of 7 years and the current one is applicable for the years 2014-2020.[^ftnt2] + +The MFF takes the form of a Council regulation, adopted following a special legislative procedure called the consent procedure: + +1. The European Commission submits a proposal for an MFF regulation; +2. The European Parliament accepts or rejects the proposal by an absolute majority vote - but it cannot amend the proposal; +3. After the approval of the European Parliament, the Council adopts the MFF Regulation by a unanimous vote. + +Through the MFF regulation, ceilings for each year concerning commitment appropriations and payment appropriations are determined for each of the MFF headings. Headings change from one MFF to the other, making comparisons difficult over the years. + +For the period 2014/2020, the headings are: + +- Smart and inclusive growth including: +- Competitiveness for growth and jobs; +- Economic, social and territorial cohesion; +- Sustainable growth: natural resources; +- Security and citizenship; +- Global Europe; +- Administration; +- Compensation. + +The framework also sets up ceilings for a number of instruments established outside the general headings and known as the Flexibility and Special Instruments, such as (1) the Emergency Aid Reserve; (2) the European Union Solidarity Fund; (3) the Flexibility Instrument;[^ftnt3] (4) the European Globalisation Adjustment Fund; (5) the Contingency Margin; (6) the Global Margin for commitments for growth and employment, in particular youth employment. These instruments enable the EU to mobilise the necessary funds to react to unforeseen events (financial crisis, emergency situations). + +One specific instrument, the European Development Fund (EDF) is managed outside the framework of the MFF and governed by its own set of rules. The EDF provides development aid to African, Caribbean and Pacific (ACP) countries, as well as to Overseas Countries and Territories (OCTs). It is based on an ACP-EU Partnership Agreement known as the “Cotonou Agreement”, which is revised every five years, and it is financed by direct contributions from EU Member States. The resources for the current EDF amount to €30.5 billion for the period 2014 - 2020. + +### What are Commitment and Payment Appropriations? + +In the EU annual budget and in the MFF regulations, authorised expenditure amounts are expressed in both **commitment** and **payment appropriations**. + +- **Commitments** are legal pledges to provide finance, provided that certain conditions are fulfilled. The EU commits itself to reimburse its share of the costs of an EU-funded project when the project is completed. +- **Payments** are cash or bank transfers to beneficiaries. Today’s commitments are tomorrow’s payments.[^ftnt4] + +This means for instance that 2015 payments are actually 2014 commitments. In practice, this implies that the EU does not finance a project in advance. Someone, a Member State in most of the cases, would have to provide the funds beforehand before being reimbursed by the EU. + +The ceilings of the 2014-2020 MFF are **€959 988 million** in commitment appropriations and **€908 400 million** in payment appropriations. This represents a decrease compared to the 2007-2013 MFF, whose ceiling in commitment appropriations was up to €994 176 million. The main areas concerned by this decrease are the Cohesion Policy, under the heading Smart and Inclusive Growth (-8%) and the Agriculture and Rural Development Policy, under the heading Sustainable Growth: Natural Resources (-11%).[^ftnt5] + +The EU Annual Budget +-------------------- + +Every year, a regulation is adopted to set up the annual budget of the EU. This budget is based on the MFF and the procedure followed is the ordinary legislative procedure.[^ftnt6] One of the guiding principles for establishing the annual budget is that *“the revenue and expenditure shown in the budget must be in balance”*.[^ftnt7] + +**Figure 1: EU annual budget regulation adoption procedure**![Annual Budget.png](../img/image08.png) + +Source: European Commission | [SVG version available here](https://drive.google.com/open?id=0BxUOTiQczyxHaDBqSmdDR0FTRlE&authuser=0) + +The annual budget regulation is divided into several sections detailing, for each of the EU institutions, the relevant revenue, expenditure and staff allocated. The largest section of the regulation concerns the European Commission, which is responsible for managing the EU policies. + +For the year 2015, the budget amounts to: + +- €145 321 million in commitment appropriations; +- €141 214 million in payment appropriations (this represents an increase of 1,57% compared to the 2014 EU annual budget); +- €141 214 million in budgetary revenue.[^ftnt8] + +It is possible to amend the budget after its adoption. + +[Annual activity reports](http://ec.europa.eu/atwork/synthesis/aar/index_en.htm) are prepared every year by each Directorate-General (DG) and contain the DG’s annual accounts and financial reports. These reports are summarised in a synthesis report, sent to the European Parliament, the Council and the European Court of Auditors (ECA). + +In addition, an annual report on budgetary and financial management is published every year by the Commission.[^ftnt9]  + +In November every year, the Court of Auditors publishes public reports on the implementation of the EU budget and the EDF by the Commission. Based on these reports, the European Parliament expresses an opinion on the implementation of the budget by the European Commission. If this opinion is positive, it means that the Parliament has granted discharge to the Commission for this financial year.[^ftnt10] + +* * * * * + +- [Return to Beginning](../) +- [Prev: The EU Budget at a Glance](../overview/) +- [Next: Where Does the Money Come From?](../revenue/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/datasets.md b/examples/openspending/content/resources/eu/datasets.md new file mode 100644 index 00000000..2d5cd380 --- /dev/null +++ b/examples/openspending/content/resources/eu/datasets.md @@ -0,0 +1,19 @@ +--- +title: Annex - The Open Data Audit of EU Funds +--- + +As we have seen, detailed information on EU spending and beneficiaries managed directly by the Commission can be found on the EU FTS, but it only concerns 20% of the EU Budget. For the remaining 80%, there is no central place to go and one has to look at every external entity spending EU money. + +In order to help in that task, we have set up the **Open Data audit of EU funds**. The aim of this census is to identify all existing datasets containing information on EU funds beneficiaries, and to assess whether or not data are available as open data. + +* * * * * + +{% for dataset in site.eu_datasets %} +[{{dataset.title}}]({{ dataset.url }}) +{% endfor %} + +- [Return to Beginning](../) +- [Prev: Recommendations and Next Steps](../recommendations/) +- [Next: Annex - Legal Basis for the Establishment of the EU budget](../legal-basis/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/diagrams.md b/examples/openspending/content/resources/eu/diagrams.md new file mode 100644 index 00000000..8a48c045 --- /dev/null +++ b/examples/openspending/content/resources/eu/diagrams.md @@ -0,0 +1,14 @@ +--- +title: Diagrams in editable version +--- + +![](../img/image07.png) + +* * * * * + +![](../img/image00.png) + +* * * * * + +{% include_relative nav.md %} +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/executive-summary.md b/examples/openspending/content/resources/eu/executive-summary.md new file mode 100644 index 00000000..c3678dba --- /dev/null +++ b/examples/openspending/content/resources/eu/executive-summary.md @@ -0,0 +1,9 @@ +--- +title: Executive Summary +--- + + + +* * * * * + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/expenditure.md b/examples/openspending/content/resources/eu/expenditure.md new file mode 100644 index 00000000..5292dbfe --- /dev/null +++ b/examples/openspending/content/resources/eu/expenditure.md @@ -0,0 +1,41 @@ +--- +title: Who Manages and Spends the Money, and How? +--- + +The European Commission has the overall responsibility of the implementation of the EU budget, but does not directly manage all EU funds. Indeed, these funds can be spent in several different ways: + +- Through **centralised management**: the money is managed directly by the European Commission (e.g. through the selection of contractors or the award of grants) - this is **centralised direct management** - or indirectly by EU executive agencies, decentralised agencies or national agencies - this is **centralised indirect management**; +- Through **shared management** (this method accounts for around 80% of the EU budget): there is a delegation of budget implementation to EU Member States; +- Through **decentralised management**: there is a delegation of budget implementation to non-EU countries; +- Through **joint management**: there is a delegation of budget implementation to intergovernmental and international organisations. + +One fund can be spent in different ways (for example, through centralised and shared management). **For each fund, a specific regulation is adopted which defines the rules related to its implementation.** + +**** + +**Figure 3****. Budget Management Overview** + +**** + +![](../img/image05.png) + +Source: [European Commission](http://ec.europa.eu/budget/explained/management/managt_who/who_en.cfm) + +Ultimately, a large number of entities are involved in the spending of the EU budget. + +- **The Commission itself and its agencies.**In that case, money will be directly spent through grants, contracts, or administrative expenditures. +- **National and regional authorities in EU Member States;** +- **Authorities in non-EU countries** which receive financial support from the EU; +- **International organisations** such as the United Nations or the World Bank. + +As a general rule, the co-financing principle usually applies to EU spending, especially when it comes to shared management. Through this rule, part of the cost of an action or the running costs of an entity is borne by the beneficiary of the grant or by contributors other than the European Commission, with the aim to make beneficiaries responsible for the operational and financial viability of their projects. Some flexibility nevertheless exists to allow for the full financing of some projects with EU funds. + +Other rules usually applicable to funding projects with EU money are: the non-profit rule; the non-retroactivity rule and the non-cumulative rule.[^ftnt15] + +* * * * * + +- [Return to Beginning](../) +- [Prev: Where Does the Money Come From?](../revenue/) +- [Next: Where Does the Money Go?](../funds/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/footnotes.md b/examples/openspending/content/resources/eu/footnotes.md new file mode 100644 index 00000000..869ef0d4 --- /dev/null +++ b/examples/openspending/content/resources/eu/footnotes.md @@ -0,0 +1,66 @@ + +[^ftnt1]: For example, see this analysis from the Robert Schuman Foundation on recent opinion polls: [http://www.robert-schuman.eu/en/european-issues/0333-overcoming-democratic-breakdown-in-the-european-union](http://www.robert-schuman.eu/en/european-issues/0333-overcoming-democratic-breakdown-in-the-european-union)  + +[^ftnt2]: See [Council regulation n°1311/2013 of 2 December 2013 laying down the multiannual financial framework for the years 2014-2020](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2013.347.01.0884.01.ENG). + +[^ftnt3]: The Flexibility Instrument was used to finance the Juncker Plan: [http://ec.europa.eu/priorities/jobs-growth-investment/plan/docs/an-investment-plan-for-europe\_com\_2014\_903\_en.pdf.](http://ec.europa.eu/priorities/jobs-growth-investment/plan/docs/an-investment-plan-for-europe_com_2014_903_en.pdf.) + +[^ftnt4]: See [the EU Funding Glossary](http://ec.europa.eu/budget/funding/information/eu-funding-glossary_en). + +[^ftnt5]: See [the Assembly of European Regions briefing note](http://www.aer.eu/fileadmin/user_upload/MainIssues/Future_of_Europe/EU_Budget_Review/MFF-briefing-note.pdf). + +[^ftnt6]: [The ordinary legislative procedure step by step.](http://www.europarl.europa.eu/aboutparliament/en/20150201PVL00004/Legislative-powers) + +[^ftnt7]: Article 310 of the Treaty on the Functioning of the European Union (TFEU). + +[^ftnt8]: [Definitive adoption (EU, EURATOM) 2015/339 of the European Union's general budget for the financial year 2015](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L:2015:069:FULL) + +[^ftnt9]: The 2014 report is available online: + +[^ftnt10]: [Reports of the ECA.](http://www.eca.europa.eu/en/Pages/AuditReportsOpinions.aspx) + +[^ftnt11]: [Council Decision of 7 June 2007 on the system of the European Communities' own resources.](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32007D0436) + +[^ftnt12]: [2014 Council Decision on the system of own resources of the European Union.](http://register.consilium.europa.eu/doc/srv?l=EN&f=ST%205602%202014%20INIT) + +[^ftnt13]: As the UK rebate, lump sum payments and reduced VAT call rates are correction mechanisms, that is to say measures taken to compensate Member States whose contribution to the EU budget is perceived as being too high compared to their relative wealth and the benefits they get out of the EU budget. + +[^ftnt14]: [Council regulation (EU) 2015/323 of 2 March 2015 on the financial regulation applicable to the 11th European Development Fund](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2015.058.01.0017.01.ENG) + +[^ftnt15]: See for example [this presentation](http://ec.europa.eu/chafea/documents/health/calls/2013/Workshop_on_Joint_Action_2013/Financial_Managment_JA%20Workshop_December_2012.pdf). + +[^ftnt16]: [The 12 highlights for the 2014/2020 MFF.](http://ec.europa.eu/budget/mff/highlights/index_en.cfm#funding) + +[^ftnt17]: Europe 2020 is the 10-year growth strategy of the European Union for the period 2010-2020. See [http://ec.europa.eu/europe2020/index\_en.htm](http://ec.europa.eu/europe2020/index_en.htm). + +[^ftnt18]: [European Commission ESIF webpage.](http://ec.europa.eu/contracts_grants/funds_en.htm) + +[^ftnt19]: Initial amount was €74 928 million but in accordance with regulation 1303/2013 (art. 92(6) and 92(7)), parts of this amount have been transferred to the Connecting Europe Facility and to the Fund for European Aid to the Most Deprived. + +[^ftnt20]: The EAGF is the fund financing direct payments to farmers and measures regulating or supporting the agricultural market. As the EAFRD, it is managed by DG AGRI, but represents a more significant share of the EU budget (around 80% of the CAP budget). + +[^ftnt21]: Source: Brigitte Alfter, cofounder of FarmSubsidy.org and Wobbing.eu. + +[^ftnt22]: For example, concerning the exact amount of the five ESIF for the 2014-2020 period, one EU publication mentions a number of [366.9 billion euros](http://ec.europa.eu/contracts_grants/pdf/synergies_beneficiaries_en.pdf) whereas another one refers to a total of [351.8 billion euros](http://ec.europa.eu/regional_policy/sources/docgener/panorama/pdf/mag48/mag48_en.pdf). + +[^ftnt23]: It should be noted, however, that DG BUDGET, the EU Directorate-General for Budget, was responsive to requests for information sent by the authors of this report. + +[^ftnt24]: The Directive  2012/17/EU introduces a unique identifier for European companies. However, there is no provision for that unique identifier to be used in data on EU funds beneficiaries. See [http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32012L0017](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32012L0017) + +[^ftnt25]: [2012 financial regulation, Articles 34 and 35](http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2012:298:0001:0096:EN:PDF). + +[^ftnt26]: The threshold can be set by Member States, see article 12 of regulation 1306/2013: [http://ec.europa.eu/agriculture/cap-funding/beneficiaries/shared/index\_en.htm](http://ec.europa.eu/agriculture/cap-funding/beneficiaries/shared/index_en.htm). The 2014 thresholds are available [here.](http://www.google.com/url?q=http%3A%2F%2Fec.europa.eu%2Fagriculture%2Fcap-funding%2Fbeneficiaries%2Fshared%2Fthreshold_en.pdf&sa=D&sntz=1&usg=AFQjCNH5M8AxntJmIsoWgMVA4HrWN4jnAA) + +[^ftnt27]: For a good introduction to data journalism, see Gray, J., Bounegru, L. and Chambers, L. (eds.) The Data Journalism Handbook (O'Reilly Media, 2012), co-edited by Open Knowledge and the European Journalism Center and freely available [online.](http://datajournalismhandbook.org/) + +[^ftnt28]: For an in depth-review of journalism investigation on EU funds up to 2012, see: European Commission, Deterrence of fraud with EU funds through investigative journalism in EU-27, 2012, + +[^ftnt29]: ICIJ, €6 billion in subsidies fuel Spain’s ravenous fleet, October 2011, [online.](http://www.icij.org/project/looting-seas-ii/nearly-eu6-billion-subsidies-fuel-spains-ravenous-fleet) + +[^ftnt30]: ICIJ,  [methodology](http://www.icij.org/project/looting-seas-ii/subsidy-methodology). + +[^ftnt31]: European Commission, Deterrence of fraud with EU funds through investigative journalism in EU-27, p. 227,2012:[http://www.journalismfund.eu/sites/default/files/EP%20Study%20-%20Deterrence%20of%20fraud.pdf](http://www.journalismfund.eu/sites/default/files/EP%20Study%20-%20Deterrence%20of%20fraud.pdf). + +[^ftnt32]: See, James T Hamilton, Democracy's Detectives: The Economics and Impacts of Investigative Reporting (forthcoming). + +[^ftnt33]: [TFUE](http://eur-lex.europa.eu/legal-content/FR/TXT/?uri=CELEX:12012E/TXT). diff --git a/examples/openspending/content/resources/eu/funds.md b/examples/openspending/content/resources/eu/funds.md new file mode 100644 index 00000000..c006824b --- /dev/null +++ b/examples/openspending/content/resources/eu/funds.md @@ -0,0 +1,334 @@ +--- +title: Where Does the Money Go? +--- + +The European Union has a budget to pay for policies carried out at the European level and for the institutions deciding, managing and implementing these policies (the Commission and its Directorates-General, the Parliament, the agencies, the Council, etc.). + +The spending priorities are set in the MFF when defining the amount of the funds under its headings. For the 2014-2020 MFF, the priorities are creating growth and jobs and reducing economic gaps between the EU's various regions.[^ftnt16] Agriculture, rural development, fisheries and environmental protection account for a major share of the budget. + +Each heading is then split into programmes and funds which finance the implementation of EU policies. The final beneficiaries can be any person or organisation (company, research organisation, local council, etc.) from the EU Member States and beyond. + +As the programmes and funds are managed by various entities belonging or not to the EU institutions, **there is currently no central database of all the EU spending beneficiaries.**  + +**The European Structural and Investment Funds (ESIF)** + +A very large amount of the EU budget is spent through 5 funds known as the **European Structural and Investment Funds (ESIF).**Indeed, with a total of 366.9 billion euros for the 2014-2020 period, the ESIF make up more than one third of the EU’s budget. To these funds, in addition to the specific regulations regarding their implementation, applies a single set of common rules defined in the [Common Provisions regulation](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2013.347.01.0320.01.ENG), with the objectives of: + +- Establishing a clear link with the Europe 2020 Strategy;[^ftnt17] +- Improving coordination; +- Ensuring consistent implementation; +- Making access to the funds as straightforward as possible for those who may benefit from them.[^ftnt18] + +The ESIF are managed under the shared management mechanism, which provides that Members States are co-responsible for the funds and, as such, must comply with specific control and audit procedures. + +In 2014, Member States had to prepare and negotiate with the Commission so-called Partnership Agreements (strategic plans with their investment priorities covering the five ESIF). Then, they adopted Operational Programmes, which break down the investment priorities and objectives of the Partnership Agreements into concrete actions. The Operational Programmes are managed by the managing/national authorities appointed by each Member State. Applications for funding are then submitted to the national or regional authority managing the relevant programme. + +The  5 ESIF  for the 2014-2020 period are: + +- The Cohesion Fund (CF); +- The European Agricultural Fund for Rural Development (EARDF); +- The European Maritime and Fisheries Fund (EMFF) +- The European Regional Development Fund (ERDF); +- The European Social Fund (ESF). + +Below are details for each of the 5 ESIF. + +The Cohesion Fund (CF) +---------------------- + +**** + + + + +++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

    MFF heading

    Smart and Inclusive Growth - Economic, Social and Territorial Cohesion

    Funds

    Cohesion Fund

    Policy supported

    Regional and Urban Policy

    2014-2020 MFF ceiling

    €63 399 million[19]

    Managing Member States

    Bulgaria, Croatia, Cyprus, Czech Republic, Estonia, Greece, Hungary, Latvia, Lithuania, Malta, Poland, Portugal, Romania, Slovakia, Slovenia

    Managing Directorate-General

    Directorate-General for Regional and Urban Policy (DG REGIO)

    Website

    http://ec.europa.eu/regional_policy/en/funding/cohesion-fund/

    + +The aim of the CF is to reduce economic and social disparities and to promote sustainable development in Member States whose GNI per inhabitant is less than 90% of the EU average. The funding goes to activities supporting the establishment of trans-European transport networks or to projects related to energy or transport as long as they clearly benefit the environment. + +Beneficiaries:  + +- **For funds under shared management**, DG REGIO manages a website with a map providing direct links to lists of beneficiaries provided by EU Member States, but the page does not make a clear distinction between the beneficiaries of spending under the CF or under the European Regional Development Fund (ERDF): [http://ec.europa.eu/regional\_policy/en/atlas/beneficiaries/](http://ec.europa.eu/regional_policy/en/atlas/beneficiaries/) +- **For funds under centralised direct management**, a list of beneficiaries until 2013 is available in the FTS, under the heading “Cohesion Funds”: [http://ec.europa.eu/budget/fts/index\_en.htm](http://ec.europa.eu/budget/fts/index_en.htm) + +The European Agricultural Fund for Rural Development (EAFRD) +------------------------------------------------------------ + +**** + + + + +++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

    MFF heading

    Sustainable Growth: Natural Resources

    Funds

    European Agricultural Fund for Rural Development (EAFRD)

    Policy supported

    Agriculture and Rural Development

    2014-2020 MFF ceiling

    €95 577.052 million

    Managing Member States

    EU28

    Managing Directorate-General

    Directorate-General for Agriculture and Rural Development (DG AGRI)

    Website

    http://ec.europa.eu/agriculture/cap-funding/index_en.htm

    + +The EAFRD is one of the two EU funds which finance the Common Agricultural Policy (CAP), the other one being the European Agricultural Guarantee Fund (EAGF), which is not an ESIF.[^ftnt20] The CAP, the agricultural policy of the EU, was established in 1962 and last reformed in 2013 to focus on 3 priorities: (i) viable food production; (ii) sustainable management of natural resources; (iii) balanced development of rural areas throughout the EU. In this context, the EAFRD focuses on the EU’s contribution to rural development programmes and is implemented through shared management with EU Member States. + +The overall CAP budget has decreased over the years, but still represents a significant share of the EU budget: from 71% in 1984 to 39% in 2013. + +Beneficiaries: + +- **For funds under shared management,**DG AGRI manages a website with links to Member States websites providing information on beneficiaries of CAP payments (although there is no clear distinction between beneficiaries of the EAFRD and the EAGF): [http://ec.europa.eu/agriculture/cap-funding/beneficiaries/shared/index\_en.htm](http://ec.europa.eu/agriculture/cap-funding/beneficiaries/shared/index_en.htm) +- **For funds under centralised direct management,**a list of beneficiaries until 2013 is available in the FTS: [http://ec.europa.eu/budget/fts/index\_en.htm](http://ec.europa.eu/budget/fts/index_en.htm) + +The European Maritime and Fisheries Fund (EMFF) +----------------------------------------------- + +**** + + + + +++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

    MFF heading

    Sustainable Growth: Natural Resources

    Funds

    European Maritime and Fisheries Fund (EMFF). Previously called the Financial Instrument for Fisheries Guidance (FIFG - for the period 1996-2006) and the European Fisheries Fund (EFF - for the period 2007-2013)

    Policy supported

    Maritime Affairs and Fisheries

    2014-2020 MFF ceiling

    €7 404.84 million

    Managing Member States

    EU28, Luxembourg excepted

    Managing Directorate-General

    Directorate-General for Maritime Affairs and Fisheries (DG MARE)

    Website

    http://ec.europa.eu/fisheries/cfp/emff/index_en.htm

    + +The EMFF supports the implementation of the European Common Fisheries Policy (CFP) and of the EU Integrated Maritime Policy (IMP) for the period 2014-2020. The objectives of the EMFF are: i) to help fishermen in the transition to sustainable fishing; ii) to support coastal communities in diversifying their economies; iii) to finance projects that create new jobs and improve quality of life along European coasts. In order to reach this objective, 11% of the fund is managed directly by the European Commission to support EU-wide objectives in maritime and coastal affairs, whereas the remaining 89% are managed by EU Member States on the basis of co-financing. In the last case, each country is allocated a share of the total fund budget, based on the size of its fishing industry - this makes Spain the biggest beneficiary of the EMFF. + +A part of the MFF ceiling dedicated to the EMFF (€900 million) has been attributed to the establishment and management of sustainable fisheries partnership agreements with third countries. Very often criticised, these partnership agreements obey to different financial rules than the general rules applicable to the EMFF (see: [http://ec.europa.eu/fisheries/cfp/international/agreements/index\_en.htm](http://ec.europa.eu/fisheries/cfp/international/agreements/index_en.htm)). + +Beneficiaries: + +- **For funds under shared management,**DG MARE maintains a website with links to the websites of Member States providing information on beneficiaries of EMFF payments (although, for some countries, the information presented seems to be outdated): [http://ec.europa.eu/fisheries/contracts\_and\_funding/the\_european\_transparency\_initiative/index\_en.htm](http://ec.europa.eu/fisheries/contracts_and_funding/the_european_transparency_initiative/index_en.htm)  +- **For funds under centralised direct management,** a list of beneficiaries until 2013 is available in the FTS: [http://ec.europa.eu/budget/fts/index\_en.htm](http://ec.europa.eu/budget/fts/index_en.htm) + +The European Regional Development Fund (ERDF) +--------------------------------------------- + + + + +++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

    MFF heading

    Smart and Inclusive Growth - Economic, Social and Territorial Cohesion

    Funds

    Less-developed regions; More developed regions; Outermost and sparsely populated regions; Transition regions; Territorial cooperation.

    Policy supported

    Regional and Urban Policy

    2014-2020 MFF ceiling

    €187.4 million

    Managing Member States

    EU28

    Managing Directorate-General

    Directorate-General for Regional and Urban Policy (DG REGIO)

    Website

    http://ec.europa.eu/regional_policy/en/funding/erdf/ 

    + +The ERDF was established in 1975. For the 2014-2020 period, its objectives are to strengthen economic and social cohesion in the EU by correcting imbalances between its regions. To do so, it focuses on several key priority areas: (i) innovation and research; (ii) the digital agenda; (iii) support for small and medium-sized enterprises; (iv) the low-carbon economy. + +EU regions are classified into several categories (see [the map presenting this classification](http://ec.europa.eu/regional_policy/en/policy/how/is-my-region-covered/)). Depending on this classification, different rules apply: + +- In more developed regions, at least 80% of the funds must focus on at least two of the key priorities; +- This is 60% of the funds in transition regions; +- And 50% in less-developed regions. + +Implementation of the ERDF is made at the regional and national levels through the principles of shared management and of co-financing. The Commission retains a supervisory role. + +The ERDF is not established as such in the MFF and in the annual budget regulations. The money channeled through this fund comes from different budget lines, such as “Less-developed regions”, “More developed regions” or “Transition regions”. These budget lines are also used to finance other funds such as the European Social Fund (ESF), making it difficult for the public to determine the exact amount of money invested in the ERDF. + +Beneficiaries: + +- **For funds under shared management,** DG REGIO maintains a website with links to the websites of Member States providing information on beneficiaries of ERDF payments (although the data is mixed up with the one provided for the Cohesion Fund): [http://ec.europa.eu/regional\_policy/en/atlas/beneficiaries/](http://ec.europa.eu/regional_policy/en/atlas/beneficiaries/) +- **For funds under centralised direct management,**a list of beneficiaries until 2013 is available in the FTS: [http://ec.europa.eu/budget/fts/index\_en.htm](http://ec.europa.eu/budget/fts/index_en.htm) + +The European Social Fund (ESF) +------------------------------ + + + + +++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

    MFF heading

    Smart and Inclusive Growth - Economic, Social and Territorial Cohesion

    Funds

    Less-developed regions; More-developed regions; Transition regions; Youth Employment Initiative

    Policy supported

    Employment, Social Affairs and Inclusion

    2014-2020 MFF ceiling

    €86.4 million

    Managing Member States

    EU28

    Managing Directorate-General

    Directorate-General for Employment, Social Affairs and Inclusion (DG EMPL)

    Website

    http://ec.europa.eu/esf/home.jsp?langId=en

    + +The ESF is the EU’s main instrument for supporting jobs, helping people to get better jobs and ensuring fairer job opportunities. For the 2014-2020 period, priorities are to boost the adaptability of workers; improving access to employment and helping people from disadvantaged groups to get jobs. + +The ESF is implemented through the system of shared management, together with EU Member States, and according to the co-financing principle. Co-financing rates vary between 50% and 85% of the total project costs depending on the relative wealth of the region (see [the map](http://ec.europa.eu/esf/main.jsp?catId=525&langId=en) presenting the classification of the EU regions into different categories). + +The ESF is not established as such in the MFF and in the annual budget regulations. The money channeled through this fund comes from different budget lines, such as “Less-developed regions”, “More developed regions” or “Transition regions”. These budget lines are also used to finance other funds such as the European Regional Development Fund (ERDF), making it difficult for the public to determine the exact amount of money invested in the ESF. In addition, and even if it is included under a different budget heading, the Youth Employment Initiative appears to be managed under the framework of the ESF. + +Beneficiaries: + +- **For funds under shared management,** DG EMPL maintains a website with links to the websites of Member States providing information on beneficiaries of ESF payments: [http://ec.europa.eu/esf/main.jsp?catId=45&langId=en](http://ec.europa.eu/esf/main.jsp?catId=45&langId=en)  +- **For funds under centralised direct management,**a list of beneficiaries until 2013 is available in the FTS: [http://ec.europa.eu/budget/fts/index\_en.htm](http://ec.europa.eu/budget/fts/index_en.htm) + +**** + +* * * * * + +**** + +**Figure 5: Share of the EU budget per headings and funds (2015 Appropriation Commitments)** + +**** + +![](../img/image02.png) + +Source of the data: [European Commission](http://ec.europa.eu/budget/figures/interactive/index_en.cfm) -  Made with Many Eyes (IBM) + +**** + +* * * * * + +**** + +**Figure 4: Members States contributors vs. beneficiaries (budget 2013)** + +![](../img/image04.png) + +Source of the data: [European Commission](http://ec.europa.eu/budget/figures/interactive/index_en.cfm) - Made with raw.densitydesign.org + +* * * * * + +All Funds +--------- + +{% for fund in site.eu_funds %} +[{{fund.title}}]({{ fund.url }}) *{{ fund.objectives }}* +{% endfor %} + +- [Return to Beginning](../) +- [Prev: Who Manages and Spends the Money, and How?](../expenditure/) +- [Next: How Transparent is the EU Budget?](../transparency/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/glossary.md b/examples/openspending/content/resources/eu/glossary.md new file mode 100644 index 00000000..27a322ed --- /dev/null +++ b/examples/openspending/content/resources/eu/glossary.md @@ -0,0 +1,65 @@ +--- +title: Glossary +--- + +Most of these definitions are taken from the [EU Glossary](http://ec.europa.eu/budget/explained/glossary/glossary_en.cfm) and from the publication [How the European Union works.](http://bookshop.europa.eu/en/how-the-european-union-works-pbNA3212336/) + +**** + +**Budget**         Annual financial plan, drawn up according to budgetary principles, that provides forecasts and authorises, for each financial year, an estimate of future costs and revenue and expenditures and their detailed description and justification. In the EU, it takes the form of a **regulation**. + +**Budget Data        **Information used in the budgetary cycle - including planning, revenues and expenditures. + +**Ceiling        **Limits of expenditure or revenue fixed by law or by **regulation** such as in the own resources decision or in the** multiannual financial framework**. + +**Commitment        **Legal pledges to provide finance, provided that certain conditions are fulfilled. + +**Consent Procedure        **Through this procedure, the** Council** has to obtain the **European Parliament**’s assent before certain decisions are taken. The Parliament may accept or reject the **Council** proposal but cannot amend it. + +**Council        **The institution where ministers of EU Member States meet to discuss EU matters, take decisions and pass laws. Which ministers attend which council meeting depends on the subject on the agenda. The Council adopts the **budget** jointly with the **European Parliament**. It should not be confused with the European Council, which brings together Heads of States or Governments. + +**Discharge (Granting Discharge)        **The decision by which the **European Parliament**, on the basis of the recommendation from the **Council** and the declaration of assurance provided by the **European Court of Auditors**, closes the life of an annual budget exercise. + +**European Commission        **Politically independent executive arm of the European Union that proposes legislation, policies and programmes of actions and is responsible for implementing the decisions of the **European** **Parliament** and of the **Council**. + +**European Court of Auditors        **Independent external audit institution of the European Union. It checks that the EU **budget** is correctly implemented. + +**European Parliament        **Directly elected legislative arm of the European Union. It shares authority with the **Council** over the budget. At the end of the budget procedure, it adopts or rejects the **budget** in its entirety (“**discharge**”). + +**Financial Transparency System        **Database of beneficiaries of **grants** and other aids paid out each year by the **European** **Commission**. + +**Fiscal Transparency        **The International Monetary Fund (IMF) defines fiscal transparency as the comprehensiveness, clarity, reliability, timeliness, and relevance of public reporting on the past, present, and future state of public finances. + +**Grants        **Direct financial contributions, by way of donations, from the **budget** in order to finance either an action intended to help achieve an objective part of a EU policy or the functioning of a body which pursues an aim of general European interest or has an objective forming part of a EU policy. + +**Headings        **In the Multiannual Financial Framework, they are groups of EU activities in broad categories of expenditure. + +**** + +**Multiannual Financial Framework        **Multiannual spending plan that translates into financial terms the EU policy priorities. It sets (a) limits on EU expenditure over a fixed period and thus imposes budgetary discipline; (b) annual maximum amounts (**ceilings**) of **commitments** for the main categories of expenditures (**headings**) and an overall **payments** ceilings. + +**Open Data        **Data is considered to be open if anyone can freely use, reuse and redistribute them, for any purpose, without restrictions (as per OpenDefinition.org). This includes being both technically open - the data can be downloaded online in open formats and read by software - and legally open - users have a legal right to reuse the data. + +**Open Budget Data        **Public financial information used in the budget cycle that is freely available in a machine-­readable format to use, modify and share (as per opendefinition.org). + +**Open Spending Data        **Information about public expenditure that is freely available in a machine­-readable format to use, modify and share (as per opendefinition.org) + +**Open Fiscal Data**        Information about public finance that is freely available in a machine-­readable format to use, modify and share (as per opendefinition.org). + +**Ordinary Legislative Procedure        **Main legislative procedure by which directives and **regulations** are adopted. It gives an equal weight to the **European Parliament** and to the **Council**in the decision procedure. + +**Own Resources        **Revenue flowing automatically to the European Union **budget**, pursuant to the Treaties and implementing legislation, without the need for any subsequent decision by national authorities. + +**Payment        **Cash or bank transfer to the beneficiaries. + +**Regulation**        Normative act, of general application, binding in its entirety and directly applicable in all EU Member States. + +**Spending Data        **Spending data is information about public expenditures - and crucially includes transaction level data, e.g. data about individual financial transactions. Ideally spending data should contain information on amount, payer, beneficiary, date and purpose of the transaction (e.g. a project or service). + +**Sugar Levies        **Mainly customs duties on imports of certain agricultural products from non-EU countries as well as levies paid by producers on sugar, isoglucose and insulin syrup. + +- [Return to Beginning](../) +- [Prev: Acronyms](../acronyms/) +- [Next: The EU Budget at a Glance](../overview/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/img/cover.png b/examples/openspending/content/resources/eu/img/cover.png new file mode 100644 index 00000000..5126622f Binary files /dev/null and b/examples/openspending/content/resources/eu/img/cover.png differ diff --git a/examples/openspending/content/resources/eu/img/image00.png b/examples/openspending/content/resources/eu/img/image00.png new file mode 100755 index 00000000..988c8c86 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image00.png differ diff --git a/examples/openspending/content/resources/eu/img/image01.png b/examples/openspending/content/resources/eu/img/image01.png new file mode 100755 index 00000000..c96abb46 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image01.png differ diff --git a/examples/openspending/content/resources/eu/img/image02.png b/examples/openspending/content/resources/eu/img/image02.png new file mode 100644 index 00000000..ddec11a8 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image02.png differ diff --git a/examples/openspending/content/resources/eu/img/image03.png b/examples/openspending/content/resources/eu/img/image03.png new file mode 100755 index 00000000..37af41d8 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image03.png differ diff --git a/examples/openspending/content/resources/eu/img/image04.png b/examples/openspending/content/resources/eu/img/image04.png new file mode 100755 index 00000000..ffb70b99 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image04.png differ diff --git a/examples/openspending/content/resources/eu/img/image05.png b/examples/openspending/content/resources/eu/img/image05.png new file mode 100755 index 00000000..781e8013 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image05.png differ diff --git a/examples/openspending/content/resources/eu/img/image06.png b/examples/openspending/content/resources/eu/img/image06.png new file mode 100755 index 00000000..bda70f32 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image06.png differ diff --git a/examples/openspending/content/resources/eu/img/image07.png b/examples/openspending/content/resources/eu/img/image07.png new file mode 100755 index 00000000..474eb299 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image07.png differ diff --git a/examples/openspending/content/resources/eu/img/image08.png b/examples/openspending/content/resources/eu/img/image08.png new file mode 100755 index 00000000..1354b5f5 Binary files /dev/null and b/examples/openspending/content/resources/eu/img/image08.png differ diff --git a/examples/openspending/content/resources/eu/index.md b/examples/openspending/content/resources/eu/index.md new file mode 100644 index 00000000..c7f802d7 --- /dev/null +++ b/examples/openspending/content/resources/eu/index.md @@ -0,0 +1,118 @@ +--- +title: "Where Does Europe’s Money Go?" +authors: +- Elisabeth Druel +- Pierre Chrzanowski +- Rufus Pollock +- Jonathan Gray +redirect_from: +- /research/eu/ +--- + +## A Guide to EU Budget Data Sources + +This report was researched and written by **Elisabeth Druel** and **Pierre Chrzanowski** on behalf of **Open Knowledge**, with input and guidance from **Rufus Pollock** and **Jonathan Gray**. Support was provided by the **Adessium Foundation**. + +**Open Knowledge** is a global civil society organisation dedicated to opening up public information, research and culture to benefit the lives of citizens around the world. Find out more at: [okfn.org](https://okfn.org/). + +**Adessium Foundation** aspires to a society that encourages people to live in harmony with each other and with their environments. The Foundation works to achieve a balanced society characterized by integrity, a balance between people and nature, and social harmony. Find out more at: [adessium.org](http://www.adessium.org/). + +Thanks to the following people who provided input, feedback and support at various stages: + +- **Brigitte Alfter**, Europe Editor, JournalismFund.eu and Lecturer, Roskilde University +- **Caelainn Barr**, Data Journalist, The Guardian +- **Cynthia O’Murchu**, Reporter, Financial Times +- **Nick Aiossa**, EU Policy Officer, Transparency International EU Office + +## Executive Summary + + ++ + + + + +

    The EU Budget in Numbers

    +

    +
      +
    • The EU has committed to spending over €959 988 million over the period 2014-2020.
      • +
    +
      +
    • The EU budget is divided into more than 80 funds and programmes.
      • +
    • Five of these funds, the European Structural and Investments Funds (ESIF), represent 33% of the total budget.
      • +
    • Around 6% of the budget goes to the administration of the European institutions, whereas around 94% is allocated to various European programmes.
      • +
    • There are more than 100 authorities managing EU funds. 20% of these are managed directly by the European Commission. The remaining 80% are managed by other authorities.
      • +
    + +Recent polls show that European citizens hold the EU to be inaccessible due to its complexity and lack of transparency.[^ftnt1] The lack of transparency and democratic accountability in European institutions is widely characterised as a “[democratic deficit](http://europa.eu/legislation_summaries/glossary/democratic_deficit_en.htm)”, a phrase which has become part of the EU’s official glossary. This “democratic deficit” is particularly apparent when it comes to EU public finances. + +This guide aims to help civil society organisations (CSOs), journalists and others to navigate the vast landscape of documents and datasets about the EU’s fiscal affairs. In doing so, our objective is to support more evidence-based journalism and advocacy, and - in the longer term - to contribute towards the transparency, public understanding and democratic accountability of EU public finances. + +The level of transparency about EU public money is highly variable, and is largely dependent on which authority is responsible for managing and disbursing a given fund. Funds which are exemplary for their transparency exist alongside funds which are effectively dark. For example, on one hand, the Financial Transparency System (FTS) set up by the European Commission provides access to granular open data about spending, but only concerns about 20% of total spending. On the other hand, information about public money spent by EU Members States is often unavailable to the public. + +The European Union has now entered into a new budget framework for 2014-2020. This is accompanied by a new financial regulation, defining new funds and new transparency rules. However, as we shall see in this report, many barriers still remain. In addition to providing an overview of key funds and programmes, we also conclude with some analysis and recommendations for further work in this area, which are summarised below. + + + + ++ + + + + +

    Summary of Recommendations

    +

    +
      +
    1. Establish a single central point of reference for data and documents about EU revenue, budgeting and expenditure and ensure all the information is up to date  at this domain (e.g. at a website such as ec.europa.eu/budget). At the same time, ensure all EU budget data are available from the EU open data portal as open data.
      2. +
    +

    +
      +
    1. Create an open dataset with key details about each EU fund, including name of the fund, heading, policy, type of management, implementing authorities, link to information on beneficiaries, link to legal basis in Eur-Lex and link to regulation in Eur-Lex.
      2. +
    +

    +
      +
    1. Extend the Financial Transparency System to all EU funds by integrating or federating detailed data expenditures from Members States, non-EU Members and international organisations. Data on beneficiaries should include, when relevant, a unique European identifier of company, and when the project is co-financed, the exact amount of EU funding received and the total amount of the project.
      2. +
    +

    +
      +
    1. Clarify and harmonise the legal framework regarding transparency rules for the beneficiaries of EU funds.
      2. +
    +

    +
      +
    1. Support and strengthen funding for civil society groups and journalists working on EU public finances.
      2. +
    +

    +
      +
    1. Conduct a more detailed assessment of beneficiary data availability for all EU funds and for all implementing authorities - e.g., through a dedicated “open data audit”.
      2. +
    +

    +
      +
    1. Build a stronger central base of evidence about the uses and users of EU fiscal data - including data projects, investigative journalism projects and data users in the media and civil society.
      2. +
    +

    + +## Table of Contents + +- [The EU Budget at a Glance](./overview/) +- [How is the EU Budget Set Up?](./budget/) +- [Where Does the Money Come From?](./revenue/) +- [Who Manages and Spends the Money, and How?](./expenditure/) +- [Where Does the Money Go?](./funds/) +- [How Transparent is the EU Budget?](./transparency/) +- [Journalistic Investigations and Transparency Projects to Follow the Money](./research/) +- [Recommendations and Next Steps](./recommendations/) +- [Annex - The Open Data Audit of EU Funds](./datasets/) +- [Annex - Legal Basis for the Establishment of the EU budget](./legal-basis/) +- [References](./references/) +- [Acronyms](./acronyms/) +- [Glossary](./glossary/) + +## Download the PDF version + + + + + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/legal-basis.md b/examples/openspending/content/resources/eu/legal-basis.md new file mode 100644 index 00000000..8654e3c7 --- /dev/null +++ b/examples/openspending/content/resources/eu/legal-basis.md @@ -0,0 +1,25 @@ +--- +title: Annex - Legal Basis for the Establishment of the EU budget +--- + +**Treaty on the Functioning of the European Union (TFEU), Title II - Financial Provisions - Articles 310 to 325:**[^ftnt33] + +Article 310: *“The Union’s annual budget shall be established by the European Parliament and the Council (...). The revenue and expenditure shown in the budget shall be in balance”.* + +Article 311: *“The Union shall provide itself with the means necessary to attain its objectives and carry through its policies”.* + +Article 312: *“The multiannual financial framework shall ensure that Union expenditure develops in an orderly manner and within the limits of its own resources. It shall be established for a period of at least five years. The annual budget of the Union shall comply with the multiannual financial framework”.* + +Article 314: *“The European Parliament and the Council, acting in accordance with a special legislative procedure, shall establish the Union’s annual budget (...)”.* + +Article 318: *“The Commission shall submit annually to the European Parliament and to the Council the accounts of the preceding financial year relating to the implementation of the budget”.* + +Article 319: *“The European Parliament, acting on a recommendation from the Council, shall give discharge to the Commission in respect of the implementation of the budget”.* + +* * * * * + +- [Return to Beginning](../) +- [Prev: Annex - The Open Data Audit of EU Funds](../datasets/) +- [Next: References](../references/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/nav.md b/examples/openspending/content/resources/eu/nav.md new file mode 100644 index 00000000..e69de29b diff --git a/examples/openspending/content/resources/eu/overview.md b/examples/openspending/content/resources/eu/overview.md new file mode 100644 index 00000000..1fde36f5 --- /dev/null +++ b/examples/openspending/content/resources/eu/overview.md @@ -0,0 +1,15 @@ +--- +title: The EU Budget at a Glance +--- + +![](../img/image07.png) + +Source: European Commission + +[SVG version available here](https://drive.google.com/open?id=0BxUOTiQczyxHY2ZBV0daZ1huMnc&authuser=0)  + +- [Return to Beginning](../) +- [Prev: Glossary](../glossary/) +- [Next: How is the EU Budget Set Up?](../budget/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/pdf/WhereDoesEuropesMoneyGo.pdf b/examples/openspending/content/resources/eu/pdf/WhereDoesEuropesMoneyGo.pdf new file mode 100644 index 00000000..24d51bcd Binary files /dev/null and b/examples/openspending/content/resources/eu/pdf/WhereDoesEuropesMoneyGo.pdf differ diff --git a/examples/openspending/content/resources/eu/recommendations.md b/examples/openspending/content/resources/eu/recommendations.md new file mode 100644 index 00000000..2bcea426 --- /dev/null +++ b/examples/openspending/content/resources/eu/recommendations.md @@ -0,0 +1,31 @@ +--- +title: Recommendations and Next Steps +--- + +As this report has evidenced, the level of transparency of EU funds is not consistent and depends on who manages the fund. **The result is a two-sided system where best practices in terms of transparency stand alongside closed data.** On one hand, the FTS of the European Commission delivers access to real expenditures information following open data standards, but it only concerns about 20% of EU spending. On the other hand, data on money spent through Members States is often unavailable to the public, and if available, not accessible in different languages. Furthermore, the release of EU Budget data as open data is still not a default practice. + +The European Union has now entered a new MFF for 2014 -2020, with the adoption of a new financial regulation, defining new funds and transparency rules. However, structural barriers still remain, preventing an effective transparency of all EU funds and public accountability of all managing authorities. + +Therefore, based on the EU budget mechanisms analysis, related transparency rules, and a preliminary assessment of effective EU budget and funds data availability, this report recommend the following actions to improve EU budget transparency and accountability: + +1. **Establish a single central point of reference**** ****for EU Budget information** and ensure all up to date documents and data are linked under this domain (e.g. at a website such as [ec.europa.eu/budget](http://ec.europa.eu/budget)). In the same time, **e****nsure all EU Budget data are available from the EU open data portal as open data.** + +1. **Set up a unique dataset of EU funds available as open data to improve the overall transparency of the EU budget.**The dataset could include the following information: name of the fund, heading, policy, type of management, implementing authorities, link to information on beneficiaries, link to legal basis in Eur-Lex and link to regulation in Eur-Lex. + +1. **Extend the Financial Transparency System to all EU funds by centralising or federating detailed data expenditures from Members States, non-EU Members and international organisations.** Data on beneficiaries should include, when relevant, a unique European identifier of company, and when the project is co-financed, the exact amount of EU funding received, the total amount of the project and all other co-financing sources. + +1. **Clarify the legal framework regarding the rules applicable to the transparency of beneficiaries of EU funds and ensure its consistency across the European Union.** Although this report did not conduct a comprehensive legal review of EU budget transparency, it has found that: (i) the differences of transparency rules applying to the beneficiaries of funds managed under centralised and shared management are preventing the establishment of a common transparency standard for all EU expenditures. This should concern also the publication, by the European institutions and in open data format, of more detailed expenditure information under the “Administration” heading; (ii) the differences between data protection law and effective enforcement within each Member State are also preventing a high level of EU funds transparency across Europe; (iii) the interpretation of the principle of proportionality regarding data protection is still subject to discussion; (iv) the transparency rules adopted by third countries and international organisations managing EU funds may also differ from the ones adopted by the EU and its Member States and prevent the establishment of a common transparency framework. + +1. **Support and strengthen civil society and journalists capacities on EU funds, together with the development of common data and investigation tools.** Newsrooms are too often reticent to launch long data investigations without the guarantee that these will pay off with good stories. In the wake of JournalismFund.eu, there is a need to support funding for data skills and relevant tools to develop data journalism and investigation projects related to EU funds.[^ftnt32] This should include not only literacy with datasets, but also literacy with data infrastructures, institutions and policies. Projects of interest in this area may include issues such as contracts/tenders financed or co-financed through the EU budget, in order to determine what are the regulations governing their functioning, what level of information is currently open to public scrutiny and what are the loopholes which may eventually be corrected in this system both to facilitate access to the data and to prevent fraud or misuse of the funds. + +1. **Conduct an assessment of EU funds beneficiary data availability (Open Data audit of EU funds).**** **Data on EU funds beneficiaries are spread into multiple data system managed by various implementing authorities such as EU institutions, Member States and international organisations. The open data audit of EU funds seeks to map all this data management system, the data they contain (from which EU funds), and their level of openness against a specific set of criteria such as the ones used in the [Open Data Index](http://index.okfn.org/). The Open Data audit of EU funds would be an essential tool for any further investigation into EU funds data as well as a powerful benchmark for any advocacy effort towards greater transparency in Europe and to assist official accountability initiatives and public institutions in identifying transparency gaps. + +1. **Build a stronger central base of evidence about the uses and users of EU fiscal data - including data projects, investigative journalism projects and data users in the media and civil society.**As well as contributing to a richer empirical picture of how fiscal data is actually used, we would like to see this work informing the development of a framework to highlight how information about EU public finances can be used to improve democratic accountability - including the different “accountability paths” through which this information can be put to work in society. + +* * * * * + +- [Return to Beginning](../) +- [Prev: Journalistic Investigations and Transparency Projects to Follow the Money](../research/) +- [Next: Annex - The Open Data Audit of EU Funds](../datasets/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/references.md b/examples/openspending/content/resources/eu/references.md new file mode 100644 index 00000000..21f8883f --- /dev/null +++ b/examples/openspending/content/resources/eu/references.md @@ -0,0 +1,36 @@ +--- +title: References +--- + +European Commission (2013). Multiannual Financial Framework 2014-2020 and EU Budget 2014 in a nutshell. Available at: [http://ec.europa.eu/budget/news/article\_en.cfm?id=201401281520](http://ec.europa.eu/budget/news/article_en.cfm?id=201401281520)  + +European Commission (2013). The European Union Explained — How the European Union Works. Available at: [http://bookshop.europa.eu/en/how-the-european-union-works-pbNA0414810/](http://www.google.com/url?q=http%3A%2F%2Fbookshop.europa.eu%2Fen%2Fhow-the-european-union-works-pbNA0414810%2F&sa=D&sntz=1&usg=AFQjCNGNF0rng49wWTbq3BnqZdFY8bfC4Q)  + +Official Journal of the European Union (2013). Council Regulation (EU, EURATOM) No 1311/2013. Available at: [http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2013:347:0884:0891:EN:PDF](http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2013:347:0884:0891:EN:PDF)  + +European Parliament (2012). Deterrence of fraud with EU funds through investigative journalism in EU-27, 2012. Available at: [http://www.journalismfund.eu/sites/default/files/EP%20Study%20-%20Deterrence%20of%20fraud.pdf](http://www.journalismfund.eu/sites/default/files/EP%20Study%20-%20Deterrence%20of%20fraud.pdf)  + +## Official sources of EU budget data + +For the source of EU funds beneficiaries, please refer to the [Open Data audit of EU funds](https://docs.google.com/spreadsheets/d/1tkKxRlkW60-ylxdvxGkMlMq8BS4SRPR4QoEd72qgFwQ/edit#gid=2028897927) + +**EU Budget 2006 - 2013 (revenue and expenditures)** + +[http://ec.europa.eu/budget/revexp/revenue\_and\_expenditure\_files/data/revenue\_and\_expenditure\_en.xls](http://ec.europa.eu/budget/revexp/revenue_and_expenditure_files/data/revenue_and_expenditure_en.xls) + +**Multiannual Financial Framework 2014 - 2010** + +[http://ec.europa.eu/budget/mff/lib/data/MFF2014-2020.xls](http://ec.europa.eu/budget/mff/lib/data/MFF2014-2020.xls) + +**Cohesion Policy Data** + +Information,** available as open data**, about the investments made and the results obtained under the Cohesion Policy (as well as contextual data on the socio-economic situation in each EU country). + +[h](https://cohesiondata.ec.europa.eu/)[ttps://cohesiondata.ec.europa.eu/](https://cohesiondata.ec.europa.eu/) + +* * * * * + +- [Return to Beginning](../) +- [Prev: Annex - Legal Basis for the Establishment of the EU budget](../legal-basis/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/research.md b/examples/openspending/content/resources/eu/research.md new file mode 100644 index 00000000..e5ea52b0 --- /dev/null +++ b/examples/openspending/content/resources/eu/research.md @@ -0,0 +1,97 @@ +--- +title: Journalistic Investigations and Transparency Projects to Follow the Money +--- + +There is a growing network of investigative journalists in Europe, supported by initiatives such as the JournalismFund.eu, who are acquiring skills to work on complex data projects. In parallel, a number of data-savvy analysts, activists and developers, are supporting investigations efforts by extracting (scraping), gathering and releasing EU budget data in a more useful way and developing dedicated tools and platforms. Those practices are often described as data journalism.[^ftnt27] Investigations can also be conducted or initiated by EU public authorities which have the mandate to control the EU Budget. The following sections present a brief and illustrative overview of a selection of recent work.[^ftnt28] + +Investigations into EU Funds +---------------------------- + +**EUobserver** + +The EUobserver is a non-profit independent online newspaper established in Brussels. It regularly covers [EU funds attribution, fraud and misuses](https://euobserver.com/search?q=EU+funds). In 2013, the EUobserver investigated [EU funds attribution to regions](https://euobserver.com/regions/119867). The result was that the poorer and more mismanaged a region was, the less it was profiting from EU funds, therefore raising the point that there is a relationship between administrative capacity and good and effective use of funds. + +**EU Transparency** + +EU Transparency is a non-profit organisation based in the UK and working to improve transparency of the European Institutions. EU transparency was behind the project FishSubsidy (see next section) and host of the Farmsubsidy.org project for some years. + +**European Research Centre for Anti-Corruption and State-Building** + +The European Research Centre for Anti-Corruption and State-Building (ERCAS) is an academic research center located at the [Hertie School of Governance](http://www.hertie-school.org/) in Berlin, Germany.** **ERCAS** **has** **notably established the [anti-corruption project database](http://www.againstcorruption.eu/?post_type=projects), where anti-corruption initiatives are documented and promoted through a database of projects, impact analysis and diagnosis reports. ERCAS also investigates itself EU funds abuse and [conducted a project in Bulgaria](http://www.againstcorruption.eu/projects/effective-investigation-of-european-union-funds-abuse-in-bulgaria/). + +**Financial Times** + +The online version of the Financial Times has [a dedicated section](http://www.ft.com/intl/topics/themes/EU_Structural_Funds) on investigations and articles on the EU Structural Funds where it reports about EU funds management issues, frauds and funds misuse. The last article raised the issue of UK local councils, London excepted, not being allowed to independently manage their share of the ESIF funds. The Financial Times has also worked with the Bureau of Investigative Journalism to make EU Structural Funds more public. (see The Bureau of Investigative Journalism and data project description in next section). + +**JournalismFund.eu** + +The [JournalismFund.eu](http://journalismfund.eu/) was founded in 2008 on the model of the [Belgian Pascal Decroos Fund](http://www.fondspascaldecroos.org/en) and is the only European fund providing working grants for journalism investigation across the EU. The fund also supports the [Dataharvest - European Investigative Journalism Conference](http://www.journalismfund.eu/dataharvest-conferences) which has become the main annual conference for journalists to share best practices and story ideas related to EU data.  + +**The Bureau of Investigative Journalism** + +The Bureau of Investigative Journalism (TBIJ) is a non-for-profit news organisation based in London. It has worked with the Financial Times on the [EuFunds project](https://www.thebureauinvestigates.com/category/projects/eu-structural-funds/) to investigate EU Structural Funds. One of the main findings from that investigation is that [only 5% (54) of 1046 leaks](https://www.thebureauinvestigates.com/2012/07/07/eu-member-states-fail-to-report-corruption-but-bureau-investigation-hits-home/) relating to corruption in 2011 came from Member States authorities, raising concerns over the shared management of EU funds. + +**The International Consortium of Investigative Journalists** + +[The International Consortium of Investigative Journalists (ICIJ)](http://www.icij.org/) is a global network of about 200 investigative journalists who collaborate on investigative stories across the globe. The ICIJ was launched as a project of the Center for Public Integrity. In 2012, the ICIJ published *“*[Looting the seas II](http://www.icij.org/projects/looting-seas-ii)*”*, an in-depth investigation into EU subsidies supporting Spanish fisheries. The investigation evidences that *"decades of overfishing have left Europe’s fish stocks in peril and its fishermen in poverty. It’s an impasse paid for by EU taxpayers."* Among the key findings, it has been found that one Spanish ship owner received more than €8.2 million in subsidies even while his company or its affiliates faced more than 40 accusations of illegal fishing and US\$5 million in fines.[^ftnt29] ICIJ used various data sources for its investigation, including data from FishSubsidy.org (see section below) but also had to requests additional information to the EU and the national and regional governments in Spain.[^ftnt30] + +**Transparency International EU Office** + +As part of the global Transparency International, the Transparency International EU Office (TI-EU) focuses on EU advocacy. It promotes accountability, transparency and integrity at the EU level. TI-EU recognised that the shared management system of a large number of EU funds makes it very difficult to access data, and is therefore [calling](http://www.transparencyinternational.eu/focus_areas/eu-budget/) for all EU funds to be available on a single, citizen-friendly website. + +Investigations can also be conducted or initiated at the EU level by dedicated authorities. + +**European Anti-Fraud Office (OLAF)** + +[OLAF](http://ec.europa.eu/anti_fraud/index_en.htm)is  charged by the [E](http://en.wikipedia.org/wiki/European_Union)U of the protection of the financial interests of the EU budget. It investigates fraud, corruption and serious misconduct within the European institutions, and develops an anti-fraud policy for the European Commission. OLAF's mandate covers all EU expenditures and it cooperates with national authorities for funds under shared management. OLAF publishes an annual report containing statistics on fraud and examples of cases investigated. According to [the 2013 repor](http://ec.europa.eu/anti_fraud/about-us/reports/olaf-report/index_en.htm)t, the main amounts of fraud were in structural funds (€111.7 million), external aid (€100.4 million), customs and trade (€84.9 million), new financial instruments (€66.3 million) and agricultural funds (€34.4 million). + +**European Ombudsman** + +The [Ombudsman](http://www.ombudsman.europa.eu/atyourservice/whocanhelpyou.faces#/page/4) is an independent watchdog with the power to investigate EU institutions. It regularly conducts investigation into EU funds. For instance, on 21 May 2014, the European Ombudsman has launched [an investigation](http://www.ombudsman.europa.eu/en/press/release.faces/en/54420/html.bookmark)on the EU [Cohesion Policy](http://ec.europa.eu/regional_policy/what/future/index_en.cfm), following concerns that substantial funding is being used to breach the fundamental rights of EU citizens with disabilities. + +**The European Court of Auditors** + +[The European Court of Auditors](http://www.eca.europa.eu/) is the independent external auditor of the EU institutions. Its role is to check that EU funds are correctly and efficiently managed and raised and spent in accordance with the relevant rules and regulations. In particular, the court checks if projects funded through direct and shared management are in line with EU priorities. + +**Project ARACHNE**  + +[ARACHNE](http://admin.interact-eu.net/downloads/7857/Presentation_) is a project from the European Commission which aims at providing Member State authorities involved in the management of the structural funds with an operational tool to identify potentially risky projects. The application collects and process data from Members States EU-funded projects, together with financial data on companies (data retrieved from [ORBIS](http://www.bvdinfo.com/en-gb/our-products/company-information/international-products/orbis?gclid=CPq2-d_am8UCFeoSwwodZVcACg)), compliance data such as list of fraudsters (data from [World Compliance, Lexis Nexis](http://www.lexisnexis.com/risk/intl/en/)) and web mining data from news articles (data retrieved from [Moreover](http://www.moreover.com/)). The project managed by DG EMPL is already in operation Bulgaria, Czech Republic, Hungary, Poland, Portugal, Romania and Slovakia. The tool's dashboard is available to Member States authorities and the Commission. However, there are as yet no plans to make the data publicly available. + +Tools for EU Funds Data Investigation +------------------------------------- + +**AsktheEU.org** + +Ask the EU is a portal managed by [Access Info Europe](http://www.access-info.org/)  which assists any citizen requesting an information from the EU institutions under the access to information right. The portal makes every request and response publicly available online, allowing for an analysis of information request on EU budget and spending. For example there are [150 requests related to “spending](http://www.asktheeu.org/en/search/spending/all?commit=Search&utf8=%E2%9C%93)[” data](http://www.asktheeu.org/en/search/spending/all?commit=Search&utf8=%E2%9C%93) and [393 requests related to “budget](http://www.asktheeu.org/en/search/budget/all?commit=Search&utf8=%E2%9C%93)[” data](http://www.asktheeu.org/en/search/budget/all?commit=Search&utf8=%E2%9C%93). + +**FarmSubsidy.org**  + +[FarmSubsidy](http://farmsubsidy.openspending.org/) seeks to obtain detailed data relating to payments and recipients of farm subsidies in every EU member state and make this data available in a way that is useful to European citizens. This is a  project of [JournalismFund.eu](http://journalismfund.eu/), a non-profit organisation. The website is powered by OpenSpending of Open Knowledge and has previously been run by EU Transparency and DICAR (Danish International Center for Analytical Reporting). The website [lists](http://farmsubsidy.openspending.org/news/media/) over 120 stories which came out from the FarmSubsidy database. + +**FishSubsidy.org** + +Similar to FarmSubsidy, FishSubsidy was a website providing detailed data related to payments and recipients of fish subsidies in every EU Member State. The website was closed down due to lack of funding. However, [related datasets are still available online](https://docs.google.com/spreadsheets/d/1NZ9tc_JN8psXXOhCHmdzsVqhvGWsE7igDf004gHgfgQ/edit?hl=en_US#gid=4) and their T[witter account](https://twitter.com/fishsubsidy) is still active. A report of the European Parliament from 2012 on investigative journalism estimated that around 40 stories came out from the FishSubsidy data.[^ftnt31]  + +**EuFunds** + +[EuFunds](http://eufunds.ftdata.co.uk/) is an initiative from the Financial Times and [The Bureau of Investigative Journalists](http://www.thebureauinvestigates.com/) (TBIJ) to collect data on beneficiaries from EU Structural Funds. The application, still online, provides a search engine to query EU structural funds data. The data were extracted between April 2010 and September 2010 and represent a snapshot of projects that have been allocated EU funds from the start of the current funding phase in 2007 onwards. The underlying database is also available as a [MySQL](http://dl.dropbox.com/u/39827429/EU_Structural_Funds/sql/euFunds.sql) and [CSV file](http://dl.dropbox.com/u/39827429/EU_Structural_Funds/csv/euFunds.csv). + +**OpenTED** + +[OpenTED](http://ted.openspending.org/) is an initiative which aims to make information from the joint European procurement system more easily accessible to journalists and researchers. The initiative was developed in the frame of the Dataharvest.eu conference in collaboration between independent journalists, developers and researchers. The dataset contains procurement data from 2013 to 2015, including contracts from EU funds. + +**OpenInterests.eu** + +[OpenInterests.eu](http://openinterests.eu/) is a catalogue of political and commercial actors related to the European Union. The project combines different sets of information including data from the FTS into a search engine, which can be used to quickly retrieve information about the activities of companies, people and institutions in a European context. + +**Wobbing.eu** + +[Wobbing.eu](http://www.google.com/url?q=http%3A%2F%2Fwobbing.eu%2F&sa=D&sntz=1&usg=AFQjCNHVQsTSB9R-DYHnv2lPmOoxuKJPgw) is a network of journalists interested in using the EU’s freedom of information legislation at national and European levels to obtain documents. The network members exchange information on developments on national and European level and assist each other with requests when necessary. Annual meetings in the Wobbing.eu network have been merged into the annual Dataharvest.eu conference. + +* * * * * + +- [Return to Beginning](../) +- [Prev: How Transparent is the EU Budget?](../transparency/) +- [Next: Recommendations and Next Steps](../recommendations/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/revenue.md b/examples/openspending/content/resources/eu/revenue.md new file mode 100644 index 00000000..4b4fbdcb --- /dev/null +++ b/examples/openspending/content/resources/eu/revenue.md @@ -0,0 +1,40 @@ +--- +title: Where Does the Money Come From? +--- + +The sources of the EU revenue are called the EU’s “own resources”. A Council Decision on Own Resources, setting up the rules applying to their calculation, is usually negotiated at the same time as the MFF regulation. Resources are then provided on an annual basis. + +The difference with the MFF regulation is that the Council Decision has to be approved by all the Member States before it comes into force - usually through a vote in national parliaments. This explains why, as of April 2015, the Council Decision on the EU’s own resources dates back from 2007[^ftnt11] and continues to apply until the new Council Decision, adopted in May 2014 by the Council, comes into force.[^ftnt12] When the 2014 Decision will be approved by all the Member States, it will apply retroactively as of 1 January 2014. + +The EU’s own resources are divided into several categories: + +- **C****ustoms duties on EU imports**: they arise from the application of the common customs tariffs and are collected on imports from third countries at the external borders of the EU. +- **A****gricultural duties and sugar levies**: they consist mainly of customs duties on imports from non-EU countries of agricultural products subject to the common organisation of the markets and levies on sugar, isoglucose and insulin syrup. + +Customs duties, agricultural duties and sugar levies are also referred as the **EU “traditional” own resources**. + +- **The VAT-based own resources:** these are national contributions based on harmonised VAT bases for Member States. +- **The GNI-based own resources:** this revenue is based on an uniform call rate applied to the Gross National Income (GNI) of each Member State. +- **Other revenues:**it includes taxes on EU staff; contributions from non-Member States and fines imposed on companies. + +The collection of revenues does not apply uniformly to all EU Member States, as rather *ad-hoc* **correction mechanisms** also exist. The three main correction mechanisms are: + +- **The UK rebate**: the UK is reimbursed a percentage of the difference between its contribution and what it receives back from the EU budget (as of today, 66% of the difference). To compensate this loss, all other Member States have to make an additional contribution to the budget in proportion to the share they contribute to the EU’s GNI. Although controversial, the UK rebate has been maintained by the 2014 Council Decision on the EU own resources; +- **Lump-sum payments**: some countries (Denmark, the Netherlands and Sweden) benefit from gross reductions in their annual GNI contributions; +- **Reduced VAT call rates**for Germany, the Netherlands and Sweden.[^ftnt13] + +As previously noted, the European Development Fund is not concerned by these rules, as it is financed directly by the Member States and has its own financial regulations.[^ftnt14] + +* * * * * + +**Figure 2: EU own resources contribution per Member States (Budget 2013)** + +Source of the data: [European Commission](http://ec.europa.eu/budget/figures/interactive/index_en.cfm) - Made with raw.densitydesign.org![](../img/image06.png)![](images/image06.png) + +* * * * * + +- [Return to Beginning](../) +- [Prev: How is the EU Budget Set Up?](../budget/) +- [Next: Who Manages and Spends the Money, and How?](../expenditure/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/eu/transparency.md b/examples/openspending/content/resources/eu/transparency.md new file mode 100644 index 00000000..c6e5ae0d --- /dev/null +++ b/examples/openspending/content/resources/eu/transparency.md @@ -0,0 +1,86 @@ +--- +title: How Transparent is the EU Budget? +--- + +Overall Fiscal Transparency +--------------------------- + +Most of the data and documentation for the budget process and its implementation are available online. The Cohesion Policy even has its own [open data website](https://cohesiondata.ec.europa.eu) offering aggregated budget data. However, there is not a unique and primary access point where all budget documents and data are centralised or federated in as standardised way. Being able to obtain a joined-up picture of EU public finances is essential for democratic accountability. For example, journalists collaborating around the EU Farm subsidy project were initially unable to determine the total agricultural subsidies going to large multi-national corporations from the EU, as the data sources were scattered across hundreds of different local, regional and national government websites, in different formats and the European Commission denied an access for documents request for the centralised dataset.[^ftnt21] + +Indeed, even though the Commission has [a central budget section](http://ec.europa.eu/budget/index_en.cfm) on its website, this report has evidenced that is was necessary to consult many websites or webpages not referenced under the Commission website in order to understand the basic principles related to the EU budget functioning and to collect related figures. + +There are also some inconsistencies between the terms and figures published in the budget documents from different EU institutions websites.[^ftnt22] As a result, accessing, analysing and understanding the whole EU Budget is a complex task which hampers EU Budget transparency. + +One of the main issues is the missing link between the two major EU budgetary instruments (the  seven years framework and the annual budget regulation) and the funds themselves. The funds set up by the multiannual framework and the budget are implemented, directly or not, by the Directorate-Generals and Agencies of the Commission, and specific information related to these funds is therefore to be found on their respective websites. But the information and data they provide are not presented in a harmonised way, making it difficult and extremely burdensome to gather the same level of information for each of the funds.  + +In addition, whereas the Commission provides a unique website to link the beneficiaries of its spending under centralised direct management (the FTS), there is no such system for the vast majority of funds spent under shared management.[^ftnt23] Diversity of languages for the different websites of the Member States also adds complexity to find and analyse the information on beneficiaries. The lack of distinction from EU and other co-funding for a project is also an issue. Lastly, datasets on beneficiaries do not use a unique European identifier for legal entities. This  hampers the capacity of investigation and other data analysis across Europe.[^ftnt24] + +Transparency Regulations +------------------------ + +According to Article 15 of the TFUE, *“the Union’s institutions , bodies, offices and agencies shall conduct their work as openly as possible. (...) Any citizen of the Union, and any natural or legal person residing or having its registered office in a Member State, shall have a right to access to documents of the Union’s institutions, bodies, offices and agencies, whatever their medium, subject to the principles and conditions to be defined in accordance with this paragraph”*. + +To implement this principle of transparency, specific obligations were defined in successive legal instruments called “Financial regulations”. The last financial regulation, adopted in 2012, established an obligation of publication of accounts, budgets and reports through means such as the Official Journal of the European Union. There is also an obligation of publication of information on recipients of EU funds. It is stated in the financial regulation that *“where natural persons are concerned, the publication shall be limited to the name and locality of the recipient, the amount awarded and the purpose of the award”*.[^ftnt25] + +Therefore, this principle of transparency is not an absolute one, as it is the subject of *“principles and conditions”*. This was evidenced in 2010, when the Court of Justice of the European Union delivered [a judgment](http://curia.europa.eu/juris/document/document.jsf;jsessionid=9ea7d0f130de9fa4dec2274b4c12a176e3a592620a2c.e34KaxiLc3eQc40LaxqMbN4ObxaSe0?text=&docid=79001&pageIndex=0&doclang=en&mode=lst&dir=&occ=first&part=1&cid=583202) in which it rules that publishing personal data on the beneficiaries of funds deriving from the two financial pillars of the CAP was not proportionate when it comes to natural persons who are the beneficiaries of these funds. As a consequence, for a two-years period, the obligation to publish information on the beneficiaries of CAP payments was limited to **only** legal persons. In 2013, [a new regulation](http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32013R1306&from=en) was adopted, which re-established the obligation to publish information on the beneficiaries of CAP payments, except when it comes to small amounts of money received by natural persons (per default less than €1250 received in one year).[^ftnt26]  + +The 2010 ruling of the Court certainly influenced the drafting of the 2012 financial regulations. In [the Commission Delegated regulation of 29 October 2012](http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L:2012:362:FULL&from=EN), it is stated that *“as far as personal data referring to natural persons are concerned, the information published shall be removed two years after the end of the financial year in which the funds were awarded. The same shall apply to personal data referring to legal persons for whom the official title identifies one or more natural persons”*. This limitation is applicable to recipients of EU funds awarded under direct and indirect management. + +For each sector or fund, more specific and detailed rules can be adopted. For example, for the 5 ESIF, [the Common Provisions regulation](http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32013R1303&:PDF)states that: *“Member States or managing authorities shall, in order to ensure transparency concerning support from the  Funds, maintain a list of operations by operational programme and by fund in a spreadsheet data format, which allows data to be sorted, searched, extracted, compared and easily published on the internet, for instance in CSV or XML format. The list of operations shall be accessible through the single website or the single website portal providing a list and summary of all operational programmes in that Member State. (...) The list of operations shall be updated at least every six months”.* + +**** + +Beneficiaries of EU Funds + +To find information on the beneficiaries of EU funds, it is necessary to look at the type of management involved, as this has a direct implication on the place where the data will be found and on which authority (the EU itself, the Member States, etc.) will manage the publication of this data. + +**** + +**Centralised Direct Management** + +For funds awarded under centralised direct management, the financial regulation provides that the information on the beneficiary shall be published on an internet site of the EU institutions. Information shall contain the name of the recipient; the locality of the recipient; the amount awarded; the nature and purpose of the measure. + +Some rules and exceptions apply as regard to data privacy. Any information identifying a natural person shall be removed two years after the end of the financial year in which the funds were awarded, and the following information should not be released: + +- Public procurement contracts worth under a certain amount (currently €15000); +- Scholarships or other direct support paid to natural person. + +In order to comply with the financial regulation, the European Commission has set up the [Financial Transparency System (FTS)](http://ec.europa.eu/budget/fts/index_en.htm). In this public database are included data on beneficiaries of EU funds from 2007 onwards, as well as data on beneficiaries of the European Development Fund (EDF) from 2010 onwards. The financial information contained in the FTS is expressed in commitments and not in payments - it corresponds to the amounts awarded (“committed”) to beneficiaries in a given year. The data is updated at least in June the year following the financial year concerned. For example, data on the year 2014 will be published in June 2015. + +**Centralised Indirect Management** + +**** + +When the management of the fund is delegated to an executive, decentralised or national agency (indirect management), it is the responsibility of this authority to publish beneficiaries data on its website. In case the information is not published directly, the website of the authority should at least contain a link to the website where the information can be found. + +The same rules apply to the publication of data on beneficiaries under centralised direct and indirect management. + +**** + +**Shared Management** + +For information on beneficiaries under shared management, it the responsibility of Member States to make the data publicly available. + +For the 5 ESIF and according to [the Common Provisions regulation](http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:32013R1303), Member States must publish data on the name of the beneficiaries, the activity and the amount of public funding allocated. For these funds and the European Agricultural Guarantee Fund, the European Commission provides on its own website links to the websites of Member States with information of beneficiaries under shared management. This concerns beneficiaries of: + +- [the Common Agricultural Policy](http://ec.europa.eu/agriculture/cap-funding/beneficiaries/shared/index_en.htm); +- [the European Regional Development Fund and the Cohesion Fund](http://ec.europa.eu/regional_policy/index.cfm/en/atlas/beneficiaries/?lan=en); +- [the European Social Fund](http://ec.europa.eu/esf/main.jsp?catId=46&langId=en&list=0); +- [the European Maritime and Fisheries Fund](http://ec.europa.eu/fisheries/contracts_and_funding/the_european_transparency_initiative/index_en.htm).  + +**Development and Humanitarian Aid to Non-EU Countries** + +**** + +Data is published directly by the European Commission on: + +- Agreements and contracts signed in the field of humanitarian aid during the year concerned, whether financed under the EU budget or the European Development Fund (for ACP countries only): [http://ec.europa.eu/echo/node/2190](http://ec.europa.eu/echo/node/2190); +- The beneficiaries of grants and contracts awarded by EuropeAid in the field of development and cooperation: [http://ec.europa.eu/europeaid/work/funding/beneficiaries](http://ec.europa.eu/europeaid/work/funding/beneficiaries).  + +* * * * * + +- [Return to Beginning](../) +- [Prev: Where Does the Money Go?](../funds/) +- [Next: Journalistic Investigations and Transparency Projects to Follow the Money](../research/) + +{% include_relative footnotes.md %} diff --git a/examples/openspending/content/resources/gift/acknowledgements.md b/examples/openspending/content/resources/gift/acknowledgements.md new file mode 100644 index 00000000..a0547799 --- /dev/null +++ b/examples/openspending/content/resources/gift/acknowledgements.md @@ -0,0 +1,44 @@ +--- +title: Acknowledgements +--- + +## Acknowledgements + +This report was commissioned by the [Global Initiative for Fiscal Transparency](http://fiscaltransparency.net/) + +The [Global Initiative for Fiscal Transparency (GIFT)](http://fiscaltransparency.net/) is a multi-stakeholder action network working to advance and institutionalize global norms and significant, continuous improvements on fiscal transparency, participation, and accountability in countries around the world. + +Authors: + +* Lucy Chambers, +* Velichka Dimitrova, +* Rufus Pollock, + +For more information contact: + +Special thanks to: + +* Martin Tisne +* Randall Kemp +* Libby Haight +* Sanjeev Khagram +* Friedrich Lindenberg +* Vivek Ramkumar +* Laura Newman +* **the many contributors who provided sections for the report.** + +We would also like to thank the Noun Project artists: +Andrew Cameron, Denis Frezzato, John Caserta, Marwa Boukarim, Øystein W. Arbo, Pedro Lalli, Scott Lewis, SZ + +This is the first edition of the Technology for Transparent and Accountable Public Finance Report + +*First Published May 2012* + +This is a report by: + +

    +Open Knowledge Foundation
+37 Panton Street, Cambridge
+CB2 1HL
+United Kingdom
+
    diff --git a/examples/openspending/content/resources/gift/bibliography.md b/examples/openspending/content/resources/gift/bibliography.md new file mode 100644 index 00000000..8a0667b7 --- /dev/null +++ b/examples/openspending/content/resources/gift/bibliography.md @@ -0,0 +1,46 @@ +--- +title: Bibliography +--- + +## Further Resources + +* *Avila, R. et al., 2010*. **Global mapping of technology for transparency and accountability New technologies**. Technology, 133(7), pp.189-204. Available at: . + +* *Bertot, J.C. et al., 2010*. **Social media technology and government transparency**. Computer, 43(11), pp.53-59. + +* *Bilbao-Osorio, B. & Dutta, S., 2012*. **The Global Information Technology Report 2012: Living in a Hyperconnected World**, Available at: . + +* *Centre for Good Governance, 2009*. **Best Practice Notes on Social Accountability Initiatives in South Asia: Social Audit of NREGS (AP) in Andhra Pradesh**, Available at . + +* *Chipchase, J. & Lee, P., 2011*. **Mobile Money Afghanistan, Institute for Money, Technology and Financial Inclusion**, Available at . + +* *Devarajan, S. & Reinikka, R., 2003*. **World Development Report 2004: Making Services Work for Poor People**, World Bank and Oxford University Press. + +* *Gigler, B.S., Tanner, B. & Kiess, J.., 2011*. **Enhanced Social Accountability through Open Access to Data: Geomapping World Bank Projects**. Development Outreach. + +* *International Budget Partnership*, **Guide to Transparency In Public Finances**, Available at: + +* *International Budget Partnership, 2010*. **Open Budgets Transform Lives: Open Budget Survey**, Available at: . + +* *Kuriyan, Renee; Bailur, Savita; Gigler Bjorn-Soren and Park, K.R., 2011*. **Technologies for Transparency and Accountability-Implications for ICT Policy and Implementation**, (DRAFT) Open Development Technology Alliance, The World Bank, Available at: . + +* *Lennox, D.*, **M-PESA Agent Anti Money Laundering Procedure**, pp.1-9, Available at + +* *Moon, S. & Mills, Z.*, **Practical approaches to the aid effectiveness agenda, Overseas Development Institute, Working Paper 317**, Available at: + +* *Nino, E., 2010*. **Access to Public Information and Citizen Participation in Supreme Audit Institutions (SAI), World Bank Institute, Governance Working Paper Series**, Available at: + +* *Open Knowledge Foundation*, **Open Data Handbook**. Available at: . + +* *Participatory Budgeting Unit*, **The role of new technology in Participatory Budgeting**, Available at: . + +* *Ribeiro, M.M., Vaz, J.C. & Matheus, R.*, **Transparency in the Portals of Brazilian Federal Government: The cases of E-procurement Portal and Transparency Portal**. In First Global Conference on Transparency Research, Rutgers University, Newark. Available at: . + +* *Ramkumar, V., 2008*. **Our Money, Our Responsibility: A Citizens’ Guide to Monitoring Government Expenditures**, International Budget Project (IBP). + +* *Solana, M., 2004*. **Transparency Portals: Delivering public financial information to Citizens in Latin America. THINKING OUT LOUD V Innovative Case Studies on Participatory Instruments**, The World Bank, Available at + +* *The Transparency and Accountability Initiative, 2011*. **Opening government: A guide to best practice in transparency, accountability and civic engagement across the public sector**, Available at: . + +* *Wohlschlegel, K. & Baxandall, P., 2012*. **Following the Money**, Available at: + diff --git a/examples/openspending/content/resources/gift/chapter1-2.md b/examples/openspending/content/resources/gift/chapter1-2.md new file mode 100644 index 00000000..59d25702 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter1-2.md @@ -0,0 +1,17 @@ +--- +title: Context +--- + +## Context + +We are living in a technical revolution. The Internet allows almost instant sharing of information, computing power permits complex calculations, and visualisations and tools to allow better analysis and comprehension of large datasets. Technology facilitates two-directional information flow, not only from governments but also to governments, allowing them to collect meaningful and structured feedback on fiscal policies. Instead of commissioning external consultancy firms to estimate citizens’ opinions and needs, governments could use the direct input from citizens through the technological and communication tools. + +Over recent years, the world has become increasingly [“hyperconnected”](http://reports.weforum.org/global-information-technology-2012/), driven by the rise of big data, mobile technology and social media. Access to the Internet has transformed the relationships between individuals, consumers and enterprises, civil society, citizens and the state, enabling accessible and immediate services and direct citizen participation. The possibility to be interconnected and to communicate instantly has created opportunities for informing citizens and including them in the decision-making processes of governments. Moreover, the price of technology is dropping, sparking extensive access to mobile technology, even in developing countries, which brings exciting opportunities for outreach, education and feedback. + +Well-established, low-cost technologies such as the radio, SMS and print hold strong, and continue to be a stronghold for outreach and communication of messages. The open data movement, particularly opening up government information through data-portals, would most likely not have been financially feasible even a few decades ago. + +These developments hold promising new possibilities for fiscal transparency and accountability, more information, better information, new possibilities for reaching out to people who would not normally interact with government, new possibilities to collect feedback and new possibilities to present information for analysis. In this report we examine two main areas: technology for transparency and technology for accountability around public finance. We will look in particular at who builds these tools, who uses them, and who benefits from them. + +With initiatives such as the [Open Data Strategy for Europe](http://bit.ly/EUopendata) soon due to come into force, and similar emphasis on opening up government data in other parts of the world, it is prudent for governments to start to look into new ways to open up their data, to save time, money and effort in opening up a backlog of data and to get necessary workflows in place. + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter1-3.md b/examples/openspending/content/resources/gift/chapter1-3.md new file mode 100644 index 00000000..4198da7f --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter1-3.md @@ -0,0 +1,219 @@ +--- +title: Highlights, Gaps & Recommendations +--- + +## Highlights, Gaps and Recommendations + + +> “There’s a really interesting dynamic interaction between work on data availability and quality, and on building tools, visualisations etc. that work with it. Without good data it’s hard to use it for transparency and accountability, and to build good tools. But without tools to start using the data, it’s hard to discover where it needs to improve.” + +> Tim Davies - Practical Participation + +In this section, we summarise the key highlights from the various sections of the report. The highlights are arranged by section of the report and followed by a few notes on the ‘gaps’: areas which, in our opinion, merit further investigation or have thus far lacked resources or coordination to make happen. Each section concludes with ‘recommendations’: We have tried to stagger these suggestions, ranging from first steps, easily obtainable goals which are likely to be feasible in the short term, to ideal scenarios, desirable end goals, which will take more commitment and longer term planning. + +### Data Portals + +In [this section](../chapter2-intro) we examine four government data portals from across the globe. The aim is to highlight government perspectives of making fiscal data available to the public. We focus on motivations for creating the service in the first instance, what features and data are offered, successes and outcomes, challenges and areas for improvement and what resources are needed to assemble and maintain such a site. + +#### Highlights + +* Data portals help address a fundamental requirement to make data accessible and discoverable once it has been made available: If data is spread across hundreds of websites and is difficult for people or machines to find, much of its value will be lost. +* The reason many government catalogues are link sites rather than data stores is often political, not just technical: Departments want to remain in charge of their data. Local departments often have to be encouraged to give up their data and be persuaded of the value of holding data centralised resource. +* Data portals can be useful both within and outside of government. Lack of information sharing affects civil servants and policy-makers -- not just those outside of government. This was highlighted in the OpenDataBC case study, where approximately one third of traffic to the citizen-facing data portal came from IP addresses within government. +* Keep it simple: A portal can be very simple initially and developed in response to user and government demands. Moreover, today exist off-the-shelf open-source solutions: Governments are increasingly learning the benefits of open-source solutions, (flexibility, adaptability, auditability etc.) but further work needs to be done to [encourage more widespread adoption](http://open-source.gbdirect.co.uk/migration/benefit.html +). +* An explicit open data policy is not needed to start an open data portal: One can begin with datasets that are already open or which are “voluntarily” donated by departments. In the UK’s case the portal preceded, and helped catalyse, development of a formal open data policy. +* Data portals could become of the most important technological tools of the government with regard to public spending monitoring and control. + + +#### Gaps + +* Greater promotion of open-source tools for open data +* More explicit opportunities for citizen participation. Some projects highlight good examples of how participation can fit within existing government workflows. See participation section for more, largely civil-society driven projects, which may provide inspiration - or source code - for some government-led initiatives promoting direct participation. + +#### Recommendations + +
    + GIFT Actions: Promoting Open Data as the Raw Fuel for Technology +

    +
      +
    • Members of GIFT should themselves ensure - whether they are national governments or international organisations - that they publish their fiscal and other financial information in full and in machine-readable formats.
      • +
    • Promote publication of key budget information as machine-readable data. Work with organisations such as the International Budget Partnership to include a related criterion in the Open Budget Survey. Set up online webinars / workshops showing how data is used after its publication.
      • +
    • GIFT should initiate the creation of a light-weight, demand-driven standard for the release of structured expenditure information to enable its comparability between countries.
      • +
    • Work closely with the Open Government Partnership to promote release of datasets which are key to contextualising financial data, such as economic indicators or procurement data.
      • +
    • Promote the release of sub-national data, perhaps via local partners, as well as national level.
      • +
    +
    + +**First Steps** + +* Fiscal data must be openly licensed and available in machine readable format in bulk. Be strict about licensing: make sure that all datasets on the portal are released under a proper [open license](http://OpenDefinition.org/). +* Engage with your user community both within and outside government. What data do they want, in what form? How can they report issues effectively and easily? Have explicit and prominent contact links so that citizens can report issues. Even better, allow them to browse past issues or see the clarification which others have received in answer to other questions e.g. through online discussion forums. +* Engage with your provider community: civil servants and others working inside government who will be publishing data using the site. What can be done to make this process as easy and rewarding as possible? +* Be timely: particularly for financial information, having current data is key to enabling re-use with an impact. +* Open tools for open data: use free and open-source tools wherever possible. + +**Next Steps** + +* Monitor data release, quality and usage and reporting this to relevant decision-makers becomes increasingly important as the project matures. + + +### Finding and Using Data + +In [this section](../chapter3-intro) we asked the open spending data community - who come from a variety of backgrounds; research, technical, media etc., what fiscal data they require, what they want to do with it, and importantly, how easy it is to obtain and use. Through a series of structured surveys, we established the following: + +#### Highlights + +* Fiscal transparency is about more than budgets: Users will often require more than just basic budget information to hold governments to account. Fiscal transparency should therefore not be limited to making these key budget documents accessible. Company and procurement data are common requests, as are audit reports and sub-national budget information. +* We received responses to the questionnaire from eighteen countries, spread around the globe. Six respondents mentioned that they used a government transparency portal to get their data, three used FOI requests, seven collected the data themselves from various sources and two said they had not been able to acquire the data. +* ‘The current status of open data’: Feedback was mixed. Some respondents praised the ease with which they acquired the data, but in the majority of cases, it was not easy to use the data that they had obtained. Barriers included: +* Machine readability: The most common complaint was non machine-readability of data - usually the complaint was about PDFs. +* Poor quality: Frequently there are problems with the data such as lack of care for the process of creating the data, poor structure, too aggregated and inconsistencies in structures between years. +* Disappearing data or hidden data: Several users mentioned that historic data disappeared from government websites and two could not get access to the data because of paywalls. A couple of more technical users of the data said an API for the data would be useful. +* There is often a lack of supporting documentation to complement data releases, meaning that users cannot find out what is actually contained within a dataset. + +#### Gaps + +* Machine-readable data +* Data-management systems which ensure good-quality, consistent data +* Preservation of historical data + +#### Recommendations + +
    + GIFT Actions: Technology Outreach and Idea Sharing +

    +
    • Promote experience sharing between governments through workshops and forums in existing social media platforms. Governments need to share the lessons learned in using technology to promote transparency, accountability and participation across different countries.
      • +
    • Involve civil society actors who use data and services provided by governments, to review and share ideas about existing and planned initiatives. Such actors could form advisory-panels on particular topics e.g. company data.
      • +
    • Create a well-maintained index of existing initiatives both an the demand and supply side, including methods of data release, re-use and presentation, discussion or activism enabled by such information.
      • +
    +
    + +**First Steps** + +* Make data available in a machine-readable, fully disaggregated form +* Provide some adequate basic information about the released data + +**Next steps** + +* Provide of machine-readable forms of important datasets such as companies registries, contract awards and underlying tender specifications +* Provide structured, machine-readable information on budgeting process, e.g. important dates (publication dates, points in the process where feedback is possible), amendments and information on “who does what” in budgeting issues +* Provide comprehensive documentation to accompany the documents i.e. read-me files explaining what a given dataset, data-dictionaries, glossaries +* Capacity-building, connecting policy and data processing/analysis expertise +* Promote data sharing of intermediate results e.g. integrated datasets produced by researchers, journalists, advocacy organisations and developers and the creation of data commons for processed output of re-use applications + +**Ideal scenario** + +* Open-source tools used within government for production and consumption of data +* Long-term archives of data on “neutral ground” i.e. not run by states nor advocacy groups, something like the Internet Archive for data +* Integration of budget, procurement, companies and contracts, metrics/evaluation and spending information. +* Real-time access to data via an API: Note that this is not relevant in all cases, e.g. for budget information that only changes once per year. + + +### Standards for Fiscal Data + +[This section](../chapter4-intro) examined whether having standards for fiscal data could provide a solution to some of the usability issues described by the users of the data in the previous section, by looking at two internationally recognised standards, IATI and XBRL. + +#### Highlights + +* Standards enable a distributed rather than a centralised approach in data publication and use. No or very limited general fiscal data standards are currently in existence and in use. +* XBRL is establishing itself as a worldwide standard for accounting information and may grow into a standard for general ledger data. It is however complex and focused on the traditional accounting domain - especially around balance sheets - which may limit its relevance and benefits as a format for transparency purposes. + +#### Gaps + +* A standard for transaction level spending data. +* Lack of widely-adopted standards for entity identifiers, which would greatly facilitate merging and matching +* Tools to merge and annotate the data +* Training and quality assurance (validation) tools + + +#### Recommendations + +
    + GIFT Actions: Technology Review +

    +
    • Review existing projects and and solutions from the point of view of both governments and civil society actors. Civic Commons Marketplace as well as the appendix, but a larger-scale, community effort will be required to keep this up to date.
      • +
    • Based on feedback from the above refine and maintain an online, collaborative catalogue of technical and policy options (perhaps as a wiki) which can guide governments about existing solutions and experiences. +
    +
    + +**First Steps** + +* Use well-known, commonly used formats such as CSV, XLS or XML for the release of data. +* Adopt existing coding conventions for shared entities, such as countries e.g. ISO-3166 and EU NUTS codes. Publish information on the identifiers used for companies and make sure they are not entangled in IP e.g. DUNS numbers or behind a paywall. +* Publish additional information on the coding schemes used, such as functional or economic classifications, charts of account. + +**Next Steps** + +* Enter standardisation process, beginning with core taxonomies. Additional work on aligning ‘spines’ of data e.g. mapping IATI data onto COFOG compliant budgets. +* Geocode data using re-usable, openly licensed coding services to avoid inclusion of proprietary IP. +* Begin using globally unique URIs for companies and other organisations, government bodies, and projects. +* Critically evaluate the options for re-using IATI or XBRL formats rather than creating new standards specifically for transaction level spending data. Explore options for format convertibility. + +**Ideal Scenario** + +* A global spending data registry which lists spending data from around the world in a standard format - similar to the IATI registry for aid information. + + +### Where Does the Money Come From? Where Does the Money Go? and The Invisible Money + +#### Highlights: Tools for Data + +* Civil society is often cited as the target users for transparency portals, as was the case for the Sierra Leone Mining Repository. However, in reality, 65 % of the users of this platform are investors, donors and other governments, who have knowledge of complex legal terminology and sector-specifics. The level of technical knowledge required to understand this data means that similar audiences may be the main user groups for other such tools. +* Design of tools for end-users are most effective when they bear in mind less-web-savvy users or users who will have to work in lower bandwidth or with older computers. Options such as saving and printing user-generated reports could be very useful for these purposes. +* “Data is not always actionable simply because it is available.” Having the possibility to download the data does not guarantee that one would be able to use it or understand its structure and source. +* Open source tools allow rapid dissemination of approaches and the adaptation of tools to different contexts internationally. For example, the visualizations from OpenSpending have been widely and independently re-used (e.g. to power the Hungarian site [“A mi Penzünk”](http://amipenzunk.hu/) and the Czech site [BudováníStátu.cz](http://budovanistatu.cz/)). +* Information to hold individual politicians personally accountable for their financial choices is highly desirable, particularly for journalists, and could have a great impact, particularly in situations where individual politicians wield choice over discretionary funds. + +#### Highlights: Participation + +* Technology can add additional channels for participation to reach out to people who might otherwise be hard to reach, or make it easier for those who already do participate. Yet technology is not a substitute for face-to-face contact. +* Technology is not a fix-all. If a tool is to be successful, it will most likely target a specific aspect of the Participatory Budgeting-cycle. A strategy can target a particular part of the PB process rather than being a super-app that will work at every stage. +* Conversations should be tightly facilitated in the idea collection and deliberation phases to ensure that the conversation remains structured and duplicate or irrelevant ideas do not distort its clarity. Governments should also ensure that they have the response capacity to be able to deal with the potentially large amounts of feedback, which could ensue before implementing a technical solution to idea collection. + +#### Highlights: Social Auditing + +* Social audits using technology offer a possibility of close to real-time feedback and reporting. +* Technology, e.g. mobile, can help transform traditional labour-intensive social auditing practices, which gives a possibility to scale-up corruption monitoring. +* SMS technology has the possibility of reaching out to rural areas with limited Internet access. +* Many of the projects featured in this section rely on sub-national level data to function: It is at local and regional level that many of the services from which citizens benefit most tangibly are delivered e.g. infrastructure, healthcare, schooling. +* Another type of data, which could be key for stimulating citizen feedback on an individual level, is data on personal entitlements, which might motivate citizens to provide feedback on whether they received their personal allowance. + +#### Gaps + +* Raw, machine-readable data is vital. Having to extract structured data from unstructured documents (e.g. PDFs and Word documents) prior to use creates work, could introduce errors (e.g. from manual transcription) and can lead to supply-side data-quality issues, e.g. if the government faces bad data management practices. +* Very few projects are currently able to trace the difference between planned and actual expenditure. Sunlight’s work in the United States is one of the few examples of success in this area. +* Good, open-source tools for doing entity matching and extraction +* Sub-national level spending data +* Data on citizens’ personal entitlements + +#### Recommendations + +
    + GIFT Actions: Technology Sharing +

    +
      +
    • Promote the use of promising projects from the Technology Review stage. Liaise with funding bodies to ensure open-source solutions are available for all.
      • +
    +
    + +**First Steps** + +* Build public-facing tools, targeted at enabling expert oversight, e.g. by investors and other governments, who have both incentives to scrutinise this data and pre-existing knowledge to enable them to do so. +* Promote use of these tools to infomediaries, who perform the role of translating complex information for the benefit of the public +* Related datasets: Fiscal data is most useful when related core datasets (e.g. reference identifiers, geospatial information) are also available +* More and better data: Access to good quality sub-national transaction-level data on expenditure; access to data on personal entitlements + +**Next Steps** + +* Contextualise the data in a way that is suitable for a layman audience - how was it generated? What is and is not covered? +* Build tools using the data that has been released as part of the ‘first steps’, in order to allow greater understanding of information for a layman audience. For example, citable, pre-computed statistics (as long as the methodology behind generating them is clear), which have been generated by experts are extremely attractive for those looking to use them in reports and for the press. +* Closer collaboration between advocacy NGOs and journalists. + +**Ideal Scenario** + +* Finishing off the great work done so far in this area in compiling country-level reports into one place and building a sortable online database. +* Extensive capacity-building for journalists to enable them to analyze and present data. +* Social Audit Portals similar to Andhra Pradesh with near real-time access to financial data. + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter1-4.md b/examples/openspending/content/resources/gift/chapter1-4.md new file mode 100644 index 00000000..d1fe7c4e --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter1-4.md @@ -0,0 +1,106 @@ +--- +title: Key to Case Study Icons +--- + +The icons below are used as a quick overview for the projects listed in this report. You will find them arranged in a row at the top of each case study. + + +## Fiscal Scope + + + + + + + + + + + + + + +
    Spending Side + +Spending Side + +Off-Budget +
    Revenue SideSpending SideOff-Budget
    + + +## Aims of the Project + + + + + + + + + + + + + + + + + + + + +
    Publish Better Data + +Educate Citizens + +Facilitate Direct Participation + +Get Feedback to Policy Makers + +Analyse and Understand Data +
    Publish Better DataEducate CitizensFacilitate Direct ParticipationGet Feedback to Policy MakersAnalyse and Understand Data
    + + +## Technology + + + + + + + + + + + + + + + + + + + + + + + + + +
    Mobile Technology + +Web-based Technology + +Offline and Print on Demand + +Data Visualisation and Maps + +Formats and Standards + +Social Media + +Radio +
    Mobile TechnologyWeb-based TechnologyOffline and Print on DemandData Visualisation and MapsFormats and StandardsSocial MediaRadio
    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter1.md b/examples/openspending/content/resources/gift/chapter1.md new file mode 100644 index 00000000..c6026ee8 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter1.md @@ -0,0 +1,47 @@ +--- +title: Introductions +--- + +## Chapter 1: Introductions and Methodology + +### Introduction + +This report, “Technology for Transparent and Accountable Public Finance”, was commissioned by the Global Initiative for Fiscal Transparency (GIFT) in February 2012 in order to assist the GIFT in assessing the potential of technology to aid transparency and accountability in relation to governments’ fiscal activities. + +This reports provides examples of projects around the world that are using technology (web, mobile or otherwise) to further aims of fiscal transparency. We focused on projects which: + +* Publish more or better data related to fiscal processes (aid, revenues, budgets, audits, etc. — see below), +* Help understand this data through the creation of better visualisation and data analysis tools, +* Educate citizens about fiscal processes, and assist civil society organisations in promoting accountable governance, +* Facilitate direct participation in fiscal matters through participatory budgeting, citizen auditing, etc., +* Provide policymakers with complete and reliable data relevant to their work, enabling them to make better decisions. + +We have sought to find projects which reflect all stages of the fiscal process: + +* Looking at [where the money comes from](../chapter5-intro): both getting more data released and building analysis and visualisation tools in the context of revenue processes e.g. taxation, extractive industries, etc., [c] +* Monitoring [where the money goes](../chapter6-intro): presenting data about the budgeting process and getting citizens involved in fiscal processes e.g. through participatory budgeting and comparisons of planned and retrospective budgets, +* [The invisible money](../chapter7-intro): improving public understanding of state-owned or semi-state owned enterprises, off-budget information and social audit projects which verify whether official money is being concealed or is not being spent according to official plans – information which often is not published as part of current budgeting practices. + +In each case, we considered questions such as: + +* Who are the users and audiences of the project? +* What are their motivations and what skills are required? +* What are the successes, failures and limitations? + +For each project, we have highlighted strengths and weaknesses based both on our own experience of developing tools, and also by seeking feedback from civil society organisations and the technical community. This report also aims to highlight the gaps: In some instances, cutting-edge technology is being used in fields besides public finance, which may merit further exploration; in other cases, we highlight points in the budgeting cycle which are currently underserved by technical tools. + +### Methodology: + +We have selected case studies that are appropriate across: + +* Different levels of literacy and access to technology +* Different budgets +* Government-led, civil society led and citizen led initiatives + +The research was a combination of qualitative interviews and direct inputs from case study representatives through online questionnaires. + +### Overview of Projects + +![Project Overview](http://farm8.staticflickr.com/7237/7273947644_4799a7e720_o.png) + +
    Next »
    \ No newline at end of file diff --git a/examples/openspending/content/resources/gift/chapter10-intro.md b/examples/openspending/content/resources/gift/chapter10-intro.md new file mode 100644 index 00000000..03eb0153 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter10-intro.md @@ -0,0 +1,12 @@ +--- +title: Appendix +--- + + +## Final Observations and Review + +The appendix to this report is a consolidated list of projects extracted from the case studies. In addition to this are other projects, which there was not space to include in the main body of the report, that are however useful examples of tools, approaches and contacts for those interested in fiscal transparency. The list includes the name of the project and the organisation, the geographical focus - where it was implemented as well as the local and national scope, the stage in the fiscal process and the type of technology, link to the tool and a short background. You can access the list [here](http://bit.ly/TTAPF-projects). + +We would like to encourage users to help keep this list up to date - if you would like to add a project, please [email us](mailto:gift-report@okfn.org). + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter2-1.md b/examples/openspending/content/resources/gift/chapter2-1.md new file mode 100644 index 00000000..484eb6d5 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter2-1.md @@ -0,0 +1,138 @@ +--- +title: CKAN in Data.Gov.Uk +--- + +## Out of the Box Solution: CKAN in Data.Gov.UK + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    CountryUnited Kingdom
    URLdata.gov.uk
    + +OpenSpending Data.Gov.Uk + +### Background +Data.Gov.UK is the UK Government’s official open data portal. The site provides a central “way into the wealth of government data” and aims to make that data “easy to find; easy to license; and easy to re-use.” Data.Gov.UK was launched in closed Beta at the start of October 2009 and entered public beta in January 2010. The project is ongoing at the present time. + +Data.Gov.UK has been built on a combination of CKAN, the open-source data portal software developed by the Open Knowledge Foundation, and Drupal, the open-source CMS. + +Initial requirements for Data.Gov.UK were data catalog capabilities (entering, editing, listing, and searching datasets) combined with basic CMS features (site content, blog, theming etc). The use of open-source plus the use of existing components which allowed for rapid development were desired (the initial prototype was developed in less than a month). Over time a variety of new requirements have arisen, most notably some need for data storage and presentation. + +Outcomes have been very positive. The UK government is continuing to use and develop Data.Gov.UK and the site has a global reputation as a leading exemplar of a government data portal. The system has successfully handled growth from a few dozen datasets to many thousands of datasets and a concomitant growth in site traffic, and the site has played a significant enabling role in the UK government’s development of its transparency and open data agenda. + +### Available Fiscal Data +1. In 2010 the UK government committed to the ongoing release of a substantial amount of open fiscal data. Specifically, in the Prime Minister’s letter of 31 May 2010 the [Government committed to](http://www.number10.gov.uk/news/letter-to-government-departments-on-opening-up-data/): +2. Historic COINS spending data to be published online in June 2010. +3. All new central government ICT contracts to be published online from July 2010. +4. All new central government lender documents for contracts over £10,000 to be published on a single website from September 2010, with this information to be made available to the public free of charge. +5. New items of central government spending over £25,000 to be published online from November 2010. +6. All new central government contracts to be published in full from January 2011. +7. Full information on all DFID international development projects over £500 to be published online from January 2011, including financial information and project documentation. +8. New items of local government spending over £500 to be published on a council-by-council basis from January 2011. +9. New local government contracts and tender documents for expenditure over £500 to be published in full from January 2011. +10. Other key government datasets: +* Names, grades, job titles and annual pay rates for most Senior Civil Servants with salaries above £150,000 to be published in June 2010. +* Names, grades, job titles and annual pay rates for most Senior Civil Servants and NDPB officials with salaries higher than the lowest permissible in Pay Band 1 of the Senior Civil Service pay scale to be published from September 2010. + +The actual delivery of individual commitments obviously took some time, but, for example: + +1. The COINS database was released in June 2010. The COINS database is the central government database for budgetary information, used by HM Treasury to manage budgeting and outturn against budget from all departments. This data was published on at (More information here: ). +2. In November 2010 the government released, and committed to ongoing monthly release of, detailed departmental and local authority transactional spending data (all spending above £25,000 for departments and above £500 for local authorities). Departmental spending data was published onto while local authorities usually published their data onto their local website or data catalog. +3. In Autumn 2010 details of central government contracts became available online. + +In July 2011 the Prime Minister issued another [letter](http://www.number10.gov.uk/news/letter-to-cabinet-ministers-on-transparency-and-open-data/ ). In addition to reviewing performance against commitments from the previous year, the letter proposed various improvements and extensions in relation to fiscal data (note that almost all major fiscal information was now open so there was little to do in terms of new data release): + +1. All government spending data to include plain English descriptions explaining the scope and purpose of every transaction, from September 2011 +2. Every department, working with the Cabinet Office transparency team, to produce an action plan in November 2011 for improving the quality and comparability of data +3. Unique reference indicators to be introduced by DBIS and HMRC beginning in December 2011. These will enable the public to track more easily the interaction between companies and government bodies +4. Working with the purchase and payment card providers to provide a consistent method of reporting government procurement card spend data for transactions above £500 in value, so this is available for publication on departmental websites, from end September 2011. + + +### Overview of Features +Data.Gov.UK acts both as a data portal and as a home for some of the government’s information on transparency and open data (for example, minutes and notes from the Transparency Board). As a data portal, its main features are: + +1. Publish and find datasets: full data catalog with rich search capabilities +2. Store and manage data: the majority of the datasets that data. gov.uk lists are hosted elsewhere e.g. on individual departments’ websites. However, there has been the need to store and manage data and an upcoming release will see these features substantially enhanced. +3. Community and social features such as the ability for users to list applications or ideas that relate to a dataset, comment, share dataset information on social media, and subscribe to RSS/Atom feeds to be kept up to date with the latest developments. +4. Federation and Harvesting: data.gov.uk acts as the UK’s hub for geospatial metadata aggregation in relation to the EU’s INSPIRE directive and therefore harvests information on geospatial datasets from a large number of other data catalogs and hubs. +5. Geospatial: Add and manage geospatial information about a dataset, view this information on maps and incorporate into search queries. +6. Rich API: Access to all dataset information over an API (Application Programming Interface). + +While this is a very rich feature-set it should be emphasised that essentials of a successful data portal can be substantially less -- data.gov.uk itself in its original incarnation had many fewer features. A data portal in its simplest form need only have a mechanism for easily listing datasets (both in human-readable and machine-readable form) -- and datasets may “point out” to data stored on other sites (e.g. individual ministries’ or departments’ websites) rather than being stored on the portal itself (though over time, there may be a need to store data, at least for archival purposes). + +### Successes +Data.Gov.UK has won widespread recognition as an exemplar data portal and its influence has been felt widely within the international community. In the UK, it has become the online home of the UK Government’s open data and transparency efforts and the data published on the site has been widely used and reused by companies, journalists, CSOs, and citizens. + +By providing a clear, and very visible, home for UK government open data it has also played a direct role in driving forward the open data and transparency agenda -- departments were already releasing datasets onto the site voluntarily *before* there was any specific policy mandating this, and publicity and interest around the site at its early stages from developers, media and others helped to galvanize further policy developments. + +In addition, data.gov.uk has played a significant role in the development of a very clear open licensing policy for UK government data by ensuring that all datasets found on the site are under an open license -- the [Open Government License](http://www.nationalarchives.gov.uk/doc/open-government-licence/open-government-licence.htm) drafted by the UK Government’s Office of Public Sector Information (now within the National Archives). + + +### Challenges +Data.Gov.UK has seen some failures. Largely, these relate to processes around data release that are not under the direct control of that project itself (though the project could take steps to ameliorate these problems). + + +To take one example related to fiscal data and the publication of the £25k spending by departments: Because each department publishes individually, this data on government spending is spread across approximately 1000 datasets on data.gov.uk (it can actually be hard to find them all because there are so many and there is no straightforward method to search for them). In addition, not all data is published in the correct format and some data links disappear as departments move data on their website. While ultimately this is a process issue, data.gov.uk have been taking steps to help improve this: for example, by ensuring consistent tagging of datasets when they are created, automatically checking datasets on a regular basis for broken links, developing validators to ensure that data is provided in a consistent format, and developing reporting tools so that Ministers and managers can get an overview of the process. + + +### Project Resources +The project was initiated in September 2009 with a go-live data 1-month later for the first version of the site. This target was met and a closed beta started in early October 2009. In January 2010 the site was made public. + + + + + + + + + + + + + + + + + + + + + + +
    PhaseDurationFull-time human resources
    Planning and data-collection1 month3
    Design and implementation3-4 (person) months (up to closed beta stage), 8 person months (up to launch)3-6
    OngoingSince Jan 20103-6
    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter2-2.md b/examples/openspending/content/resources/gift/chapter2-2.md new file mode 100644 index 00000000..a7ec2534 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter2-2.md @@ -0,0 +1,125 @@ +--- +title: DataBC Open Data Portal of British Columbia +--- + +## DataBC Open Data Portal of British Columbia + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    CountryCanada
    URLwww.data.gov.bc.ca
    + +DataBC + +### Background +When the Province of British Columbia decided to launch the first Canadian provincial level data portal offering access to more than 2,500 datasets of local and province level data, it followed the need to address both the needs of the community and to transform and civil service system. With a large part of the workforce retiring, the accumulated knowledge and skills had to be transferred and new technology harnessed for realising three main aims: + +1. Citizen participation and supporting a community around open data, +2. Data self-service - transforming the way government is present on the web and providing automated access to information +3. Business innovation - encouraging the use and re-use of data for fostering innovation through better information and certainly in government policy. + + +### Available Fiscal Data +revenue side of public finance, pre-budget statements (projected budget), data from quasi-fiscal activities, economic indicators, executive budget proposal, enacted budget, supplementary budget + + +### Overview of Features +The BC open data portal contains data across a broad spectrum of subjects; access to tools to visualise and analyse the data; and a blog with posts related to the data. The contextualisation of the original document is accomplished through the possibility to “follow” the document, tracing its progression, and the source document link is provided alongside the dataset. The contact form encourages users to ask questions, to enquire about more data when needed, as well as to provide general or specific feedback. + + +DataBC is also trying to balance between servicing the data and supporting the already existing open data community in British Columbia, from whom they received valuable feedback for the project. They are an active Twitter user with a growing number of followers. Organising hackathons also aids developers in using the data and creating data visualisations and apps which present different ideas and communicate messages around data. + + +### Standards, Licenses and Formats + +Having used the example of the UK Government’s Open License for Public Sector Information, most of the BC data can be freely reused for commercial purposes, advocacy or personal use. + +As the portal provides access to many different types of data, the underlying format and standards vary. There is a need to separate the final format from the file structure - e.g. it could be data in CSV format but the data could be poorly-structured. DataBC had to resolve problems related to the integration of the existing data management systems within the government as no mechanisms for data sharing existed previously. + +### Successes and Outcomes + +Probably one of the less-expected impacts was seeing how civil servants themselves are using the portal, as about one third of all the traffic originates from government IPs. Technology enables faster access to relevant data within the government departments, contributing to better interconnections between people working e.g. on labour policies and requiring fiscal data on income tax revenues and unemployment benefits. The most popular data turned out to be the financial data of the government, which signals the importance of transparent public finance based on the demand for information. There was also about 20% increase in the number of Freedom of Information (FOI) requests, showing that releasing a small amount of data fuelled wider interest in data. + +
    + Transparency within government +Transparency is not just about allowing civil society access to information. Governments also need information transparency internally and civil servants often suffer from lag times in waiting for data from other departments. Probably one of the less-expected impacts of creating the BC data portal was seeing how civil servants themselves are using the portal: about one third of all the traffic originates from government IPs. +
    + +### Challenges + +Probably the “Holy Grail question” of government data portals is how the data is being used and re-used and what value it generates. As there is no authentication for datasets download and privacy policies only allows the tracking of institutional use, it is difficult to know where the data goes and how it is being used. + +It was difficult to switch from a closed system in which data publication was produced in closed formats such as PDFs to one in which the importance of data management, useful metadata etc., is prioritised. Those working on the project experienced an asymmetry in the readiness of various departments to open up their data, for example, those in geospatial data already had much experience with opening up their data, whereas to those in finance, it was the first time that they had been asked to think about their data as ‘open data’ and progress was slower. For DataBC the project was not a compliance exercise, but an attempt to integrate governance, policy-making and citizen participation. + +### Project Resources (approximate): + + + + + + + + + + + + + + + + + + + + + + +
    Phase,DurationFull-time human resources
    Planning and data-collection3 months3
    Design and implementation2 months12-20
    OngoingSince July 20118
    + +### User comments and feedback: + +#### Herb Lainchbury, open systems developer: + +> “The main things I would like to see are: + +> * more raw data - right now in DataBC we see a lot of small aggregated data tables which are not all that useful, +> * more interesting data - there are easy ways to tell what data is interesting but so far BC is not releasing much of it [...] +> * license - I would also like to see them use a license that conforms to the open definition...” + +

    Thanks for input from David Wrate, DataBC

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter2-3.md b/examples/openspending/content/resources/gift/chapter2-3.md new file mode 100644 index 00000000..641064d6 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter2-3.md @@ -0,0 +1,91 @@ +--- +title: Introductions +--- + +## D-Brain in Korea: Digital budgeting and accounting system + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    CountryRepublic of Korea
    URLwww.digitalbrain.go.kr
    Vital StatisticsTotal 77,000 registrations of central and local government officials and public agency users. 200,000 transactions by 15,000 users per day. Payment of about 3.6 trillion won ($3billion) on daily average (Korea’s 2010 budget : 292.8 trillion won). Connected to 55 related information systems.
    + +D-Brain + +The top-ranking e-Government nation according to the e-Government Development Index and UN Global [E-Government Survey](http://www2.unpan.org/egovkb/global_reports/12report.htm) was the Republic of Korea. Korea’s solution for a public facing Digital Budget & Accounting System (DBAS - or nicknamed D-brain), adopted in 2007, has also been making waves as a leading model of innovative digital budgeting. + +More than a data portal, D-Brain is a web-based participatory budgeting system, which ensures citizens’ participation throughout the entire budget cycle from budget preparation to audit. D-Brain was covered in detail by Gigler et al. (2011) in their report ‘Technologies for Transparency and Accountability: Implications for ICT Policy and Administration’ and as such, we will not replicate the same research. A couple of key points from their research are highlighted below. + +### Overview of Features (see [Gigler et al.](http://scr.bi/wb-report-draft)) + +1. Integrated web-based system providing the public with real time analysis on government’s fiscal activities including budget formulation, execution, account settlement and performance management. +2. Participatory budgeting where the central government, local governments, public institutions and the public collaboratively decide on the allocation of resources and participate in nationwide fiscal decision making. +3. Citizen participation is enabled throughout the budgeting process through internet surveys, an online bulletin board, online bidding, a cyber forum, d-budget participation corner and public hearings to name but a few. Furthermore, there is a Budget Waste Report Center which offers a hotline and on-line system for citizens to prevent central government agencies and local government offices from misbehaving. Citizens are encouraged to report alleged misappropriations of government funds and are incentivised to do so with a ‘budget saving incentive bonus’, at most $30,000 which can be awarded to a citizen reporter if allegations are found to be true. + +### Benefits for Government + +1. System enables a better use of the national budget by reducing duplicative expenditure, leading the nation towards a more efficient fiscal policy. Oversight led by both policy makers and the public, who all have access to the necessary budget information to validate the accuracy and reliability of the budgeting records. +2. Before the existence of d-Brain, Korea had experienced budgeting and accounting difficulties and no feedback mechanisms to monitor inefficiency. These difficulties included no ability to strategically distribute national resources, as information was not efficiently shared even within government departments. + +### Successes and Outcomes + +1. For the congress, it has become easier than ever to review budgeting and payment information for the different sub-ministries +2. The budget authority is also able to make accurate budgeting decisions, due to their increased ability to review financial statements of previous projects in detail. They are able to better predict each expenditure line item of a future project, by which means it can also systematically manage the financial risk of the project. + +### Challenges + +1. The government needs to assess new areas that can be linked to the system and reflect it in the system to continuously maintain the efficiency of the system. +2. Although the public participation rate has increased, it has shown that they have the tendency to remain as a passive user only making electronic payments and transfers. The public institutions will have to devise a way for them to become more active participants in the fiscal decision making process. + +### Scalability + +Some of the reasons for the success of the project are: + +1. ‘The nationwide ICT infrastructure and high ICT literacy’ - ICT training is subsidised by governments (Gigler et al.). Korea achieved the world’s second largest broadband penetration in early 2000 thanks, in part, to the active role of government in promoting it and according to the ‘Ten Million People Internet Education Project’, South Korea has the highest internet user rate as a percentage of the population. Computer literacy is also a requirement of entering college in South Korea. +2. Web-participation is common in Korea, which provides most services over the Web. The approach may seem more alien and be less successful in countries where this is not the case. +3. The initiative is supported by both the private and public sector. The Korean government steered the project, with cutting edge ICT technology provided by Samsung and LG CNS - this approach can be explored in other countries. +4. D-Brain streamlines information exchange between central government, local government and public agencies, providing real time processing between agencies and facilitating payment collection. +5. People are interested in fiscal activities, as more efficient use of funds by the government is directly linked to lowering their taxation. + +### Resources + +Total cost: $63 million + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter2-4.md b/examples/openspending/content/resources/gift/chapter2-4.md new file mode 100644 index 00000000..26514720 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter2-4.md @@ -0,0 +1,101 @@ +--- +title: Brazilian Transparency Portal - Access to Information and Citizen Oversight +--- + +## Brazilian Transparency Portal: Access to Information and Citizen Oversight + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    CountryBrazil
    URLhttp://www.portaltransparencia.gov.br/
    + + +Brazilian Transparency Portal + +### Background + + +In 2004, the Brazilian Office of the Comptroller General (CGU) created the Transparency Portal, a tool that aims at increasing fiscal transparency of the Brazilian Federal Government. Developed in partnership with the Federal Data Processing Service – SERPRO, the Transparency Portal relied on the collaboration of diverse Ministries and bodies of the Federal Public Administration to advance the transparency of information on the application of federal public resources and to offer a tool that can stimulate citizen participation. The Portal brings, in simple language, detailed information about how public resources are spent. It is known as one of the most important tools of e-government in regard to public spending control. + +### Available Fiscal Data + +#### Expenses: +1. Daily Information: Budget and financial execution data with daily updating of the acts performed by administrators, +2. Transfer of Resources: To states, municipalities, companies, transfers made in foreign land, or directly to an individual, +3. Direct Expenses by the Federal Government: Construction contracts, government purchases, per diems paid and expenses on government credit cards, indicating the person that expensed any of the financial operations above + +#### Revenue: + +1. Budget and financial execution data with daily updates of the estimated government revenue. + +#### Other + +1. 2014 World Cup and Rio 2016 Olympic Games Expenses + +### Overview of Other Resources and Features + +1. Agreements: Register of Covenants, government partnerships sealed in the past years +2. CEIS: List of companies sanctioned by public entities +3. Public Officials: List of public officials, including information about position, function and functional position, with a link to a schedule of remuneration and government positions +4. Other links: List of all institutions of the federal government that have their own transparency websites +5. Other information and Services: Guidelines about public participation and citizen control, including a section “saiba mais” (know more about it) about government programs and how to police/audit them, +6. Download of queries +7. Data in graphs and other visualizations + +### User Groups and Audiences + +The Portal registers an average of 410 000 monthly visits. The number of citizens accessing the Transparency Portal has grown from 10 000 to 336 512 per month, between 2004 and 2012. A total of 44 640 citizens are registered and receive information about resource transfer in government partnerships with 5 561 municipalities. Information is sent to registered users via email. + +### Successes and Outcomes + +There have been a number of concrete cases where the Transparency Portal has supported direct social control of government activities. At the beginning of 2008, Brazil’s domestic media published numerous reports about “suspect” expenditure made using federal government Payment Cards. In one case the reports led to the resignation of one federal minister. In other cases the portal’s data has given rise to unsubstantiated media reports. + +The Transparency Portal will only be successful if a variety of stakeholders are engaged in anti-corruption efforts. Citizens, non-governmental organizations, press, private sector, academic institutions, think tanks and other civil society actors have an indispensable role to achieve the project’s effectiveness. To improve the interaction between government and civil society, the Transparency Portal discloses a communication channel: “Contact us”. Through this channel, the Portal users can elucidate any doubts related to the accessibility or to its own content, as well as praise or make suggestions. Through this channel the Portal also receives many suggestions for improvement and denounces of misuse of public money. + +### Challenges + +Monitoring and evaluating the impact of the Transparency Portal is an evolving area. The Office of the Comptroller General of the Union measures the average time spent on the website, as well as its bounce rates, pages per visit, visitors, most demanded searches, among other data. Access numbers have been used to identify demands for different queries and the depth of information researched by users. This analysis allows web managers to prioritise the most requested information in the layout. Bounce rates can be used to improve e.g. the frequently asked question section. + +### Project Resources + +The Portal is administered by a Management Group and a Working Group (operational), composed of representatives of the areas of Corruption Prevention and Strategic Information, Internal Control and Information Systems, approximately 12 public officials (including Managers and operating team, most of them part time). In addition, there is a Technical Technology Group in total approximately 10 public officials working directly to Portal. This makes use of Public budget resources of the Federal Government. + +Approximately 137 000 USD was invested for the initial stage of the Portal’s development, from the Office of the Comptroller-General regular budget. Nowadays, the Office of the Comptroller-General of Brazil (CGU) controls the program’s database and the maintenance cost is relatively low. + +

    Thanks for input from Izabela Moreira Correa, CGU

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter2-intro.md b/examples/openspending/content/resources/gift/chapter2-intro.md new file mode 100644 index 00000000..ee124002 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter2-intro.md @@ -0,0 +1,26 @@ +--- +title: Chapter 2 - intro +--- + +## Chapter 2 - Publishing Fiscal Data: Government Perspectives + +Uploading Data + +When it comes to proactive transparency and publishing fiscal data, there are numerous considerations to be taken into account. In this section, we take a look at different approaches to government financial data portals from the perspective of the people behind them; solutions range from out of the box, ready to implement, customisable, open-source solutions to custom sites, built specifically for the part of government they serve. + +We look at: + +1. Motivations for creating the service in the first instance +2. Available fiscal data and features +3. Successes and outcomes +4. Challenges and areas for improvement +5. Resources are needed to assemble and maintain such a site + +Featured Case Studies: + +1. [Data.Gov.Uk](../chapter2-1), United Kingdom +2. [DataBC](../chapter2-2), Canada +3. [DBrain](../chapter2-3), Republic of Korea +4. [Brazilian Transparency Portal](../chapter2-4), Brazil + +
    Next »
    \ No newline at end of file diff --git a/examples/openspending/content/resources/gift/chapter3-1.md b/examples/openspending/content/resources/gift/chapter3-1.md new file mode 100644 index 00000000..7e10e2a3 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter3-1.md @@ -0,0 +1,66 @@ +--- +title: Introductions +--- + +## Testimonials: User Perspectives on using Government Fiscal Data + +The statements below all come from re-users of fiscal data outside of government. Each person talks about what fiscal data they required, what they wanted to do with it, and importantly, how easy it was to obtain and use + +### Canada: DataBC and tools for non-developers + +> “Another nice thing about the data.gov.bc.ca website is that it provides tools for non-developers, so that they can play with, and learn from, some of the data.” + +### Republic of Korea: Perspectives on D-Brain + +> “D-Brain is only open to government officials viewing data. I cannot imagine getting data of my taste (say using SQL to select variables, and do simple analysis on it) from D-Brain. It only shows numbers of each categories by keyword searching.” + +### Albania: Providing datasets to advocacy NGOs to challenge commonly accepted facts + +> “In Albania, every governmental agency has to provide activity data, but the data is not being analytically used for different reasons. Sometimes the process of the data generation in these agencies is neglected, [...] and sometimes these data are erased after a short period. [...] The Albanian government has a statistical office, the Institute of Statistics (INSTAT), which works directly under the authority of Council of Ministers. There is a limited amount of the available data being published and the credibility of INSTAT is questionable - has been accused many times (on) of hiding important data or providing ambiguous data in favour of the government.” + +### Australia: Monitoring industry influence on government + +> Problems with the Data: +> +> 1. Not openly licensed by default +> +> 2. Not good data access - we have to get weekly exports every Sunday in CSV and after 2 years they disappear from the data store, so if a project was starting today they could not get all the data. There is no programmable API to access these exports, just a web page. +> 3. No validation/verification - having to build up the database week by week means we could lose transactions and not notice +> 4. Company data is closed - In Australia, the company register is pay-per-use so we cannot afford to find out who the directors of 300,000 different records are. + +### Bulgaria: Developing a visualisation of the Bulgarian budget using OpenSpending.org + +> Interesting Documents: +> +> 1. State Budget - Published as a PDF containing the budget law - very ugly. For the OpenSpending budget visualization I made, I had to parse the consolidated financial framework. [Level of detail is not sufficient] I would like to dig down to each expense. +> 2. Public Procurement Information - Published on a website - can be parsed. [Data is very timely] no more than a day or two delay. +> 3. State Audit Agency reports - Published on a website - can be parsed. Financial data on various topics of interest like: election financing, property of the public figures, political donations, party finances and other very, very interesting datasets. [Regarding timeliness] Periodic, they are ok with the terms. +> 4. Company register - Detailed information about all companies in Bulgaria way more detailed than is public in UK. [Unfortunately] Data is behind paywall - 15 000 euros/year, there is an API included in the same 15 000 euros/year cost. [Information is] very timely, updated every moment. + +### Croatia: Institute of Public Finance + +> “MoF ([Ministry of Finance](http://www.mfin.hr/)) web has a lot of interesting data [...] probably too rich, more fit for experts than for citizens or journalists. [...] [State Auditing Office](http://www. revizija.hr/hr/) claims that it puts all their reports on their web, but they don’t keep them there.” + +### Estonia: Making budgets understandable for citizens + +>“When published, the budget structure is too complicated, even for specialists in any policy field. The format is not machine-readable. So there is access to information but barriers for its usability and re-use by non-government groups.” + +### Georgia: Increasing fact-based dialogue + +>“One of our goals was to begin to make an online interactive interface which brings Tbilisi’s expenditures down to earth in a way a larger audience can understand.[...] The data, unfortunately, only covered the first two levels of the COFOG classification system, and so lacked the detail to allow me to understand in reality how Tbilisi City Hall spent funds. The reason I did not use a government transparency portal was because at that time it did not exist. Apparently, one exists now, but it was never advertised and nobody knew about it." + +> “...it would make sense that the financial system used to create and implement Tbilisi City Hall’s budget and keep track of expenditures have a public API that would allow for real time queries that provided comprehensive, detailed, up-to-date and machine-readable results.” + +### Greece: [Visualising the Greek Annual Budget](http://projects.thodoris.net/budget/ ) + +> “We collected data by extracting them from PDFs [...] it would be ideal , if they made them available online in open - machine readable formats , either as digital files or through a web service. A similar problem occurred while we tried to collect financial figures for public debt and deficit” + +### Mexico: Applied Research into public policy + +> “The TP portal has visualizations and graphics that are useful to understand the budget process and it also has a citizen’s budget, but it lacks most of the raw [disaggregated] data you can find in the SHCP portal. The “Analíticos Presupuestarios” is a fairly good source of data but it lacks details such as the classification by “partida” (the most specific economic classification that the Mexican government use). It will be very useful if all the main budget documents have the same data structure (know it is not possible to follow some classifications in the in-year reports for example). Tools such as the [Peruvian “Transparencia Económica](http://ofi.mef.gob.pe/transparencia/Navegador/default.aspx)” are very useful to systematize specific information.” + +### Nigeria: Building a budget-cut and crowd-sourcing oversight app + +> “For sub-national data who have no defined government portal, we have been able to use our informal contacts to [get] data from two states out of thirty six states. We need to reach out to the states as most of them are not used to providing their budget data to the public. Our short term goal is to drive transparency within these states” + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter3-intro.md b/examples/openspending/content/resources/gift/chapter3-intro.md new file mode 100644 index 00000000..70124fb7 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter3-intro.md @@ -0,0 +1,28 @@ +--- +title: Using Fiscal Data - Civil Society Perspectives +--- + +## Chapter 3 - Using Fiscal Data: Civil Society Perspectives + +User + +The Open Budget Index has been instrumental in establishing the key budget documents that every government should publish. The Open Budget Index, grades governments according to their publication practices (timeliness, how many key documents are published etc.)[1]; the survey does not currently look at the file format of the released documents. + +Getting format and content right is vital to ensure data can be used to hold governments to account. In this section, we examine from a user perspective how easy it is to find, get hold of and use that data. We asked the open spending data community (who come from a variety of backgrounds; research, technical, media etc.) what fiscal data they required, what they wanted to do with it , and importantly, how easy it was to obtain and use. The examples demonstrate that simply publishing the data is not enough, attention must also be paid to how the data is published. + +Respondents were asked 6 questions: + +1. How does data on government financial processes relate to your work? What is your mission? +2. How did you get the data? +3. If you used a government transparency portal to obtain the data, was it user friendly and were you able to find and access all the information you need? +4. If your government has a transparency portal but you did not use it - please explain why. +5. Please explain any issues with the data. +6. What could be done to make your work easier? + +The [section below](../chapter3-1) contains quotes and in place paraphrased responses from some of the participants interviewed. + +
    + [1] Pre-Budget Statement, Executive’s Budget Proposal, Enacted Budget, Citizen’s Budget, (Supplementary budgets), In-Year Reports (Monthly / quarterly), Mid Year Review, Year End Report, Audit Reports. +
    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter4-1.md b/examples/openspending/content/resources/gift/chapter4-1.md new file mode 100644 index 00000000..6f5a1c52 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter4-1.md @@ -0,0 +1,120 @@ +--- +title: IATI +--- + +## International Aid Transparency Initiative (IATI) + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageInternational
    URLhttp://www.aidtransparency.net
    Users/AudiencesDonors, parliaments of developing countries, private companies and foundations
    + +IATI + + +### Background +Building on around 20 years of previous work, the first version of the International Aid Transparency Initiative (IATI) standard was launched in early 2011 and aid donors around the world started publishing to it. The IATI standard documents data about both aid donors and aid activities, enabling comparison and encouraging good practice in data management. The IATI standard also includes space to attach relevant documents and details of project results, to allow the standard to build context around the data and become an end-to-end solution, tracking projects from inception to execution. + +The IATI standard has seen far greater uptake than any previous effort in terms of organisations electing to publish their data in the standard. These successes are often attributed to the multi-stakeholder nature of the design process, including both policy makers and technical experts. From 2009 to 2010, IATI consulted with a wide range of stakeholders on the design of the technical standard, alongside a parallel process to secure donor support for publishing their aid information. + +### The Role of Technology + +In the past it was common to respond to information shortages by building a new database. But by working with open data principles, IATI allows a more distributed solution - where information can flow between organisations in many different ways, not just into a central database. + +The benefits of this approach have already been demonstrated, with many NGOs and charities choosing to follow the IATI Standard, although it was developed primarily with governments in mind. + +Donors publish aid information as a feed which can be read by many different applications; both those created by other donors, by the open data community, and - importantly - by software providers to developing country financial systems. By providing aid information in a standard format, many different users can access the data in the way they need to - and developing countries can see the resources, which are supposed to be flowing to them. + +### Main User Groups: + +1. Parliamentarians in developing countries gain a better oversight of the aid resources available. Knowing where to allocate resources in their own budgeting processes is vital to ensure that money is spent in the best way / most efficiently. Sometimes, there is a transparency-asymmetry between different parts of government. For example, a treasury may be very willing to open up the information it holds, but departments which benefit strongly from aid donations (e.g. departments of health) may be more reluctant to be transparent about aid revenues, as they will not want to 'lose out' from central government budgeting. +2. Donors: Know where their money is going and whether it is being spent correctly. +1. Private companies and foundations: e.g. [Akvo](http://www.akvo.org/), who are examining options for using IATI in their really simple reporting system (RSR). +Aid Info Labs has also done a more extensive series of profiles on potential users of [IATI data](http://www.aidinfolabs.org/archives/category/inspiration/people). + +### Does the Project Require the Users to Have Specific Skills? + +Right now, direct access to raw IATI data requires some technical skills, but tools are being created like spreadsheet export of the raw data and preview tools to lower the barriers to entry. + +By working to improve the quality of the data and working with different infomediaries, from developers to researchers to activists on the ground, we can make sure that people can get access to the information they need. + +### Project Resources + +IATI centrally has needed both political and technical skills in the team - to facilitate a standard development process with a wide range of stakeholders. This involves being able to help people come together from across the world, which takes many resources. + +There are many different donors publishing IATI data, so they’ve needed different skills and support. Big donors have had to adapt their existing databases to output IATI data. Small donors have had a range of tools available to help them create IATI data - like openaidregister or aidstream. + +To use the data, some technical skills are needed, but also, importantly, skills to understand the complex aid infrastructure are important. AidInfo have been developing a one week course in understanding aid which will be piloted with NGOs in Nepal later this year, and which will be delivered alongside access to technical support to use IATI data provided by Young Innovations Nepal (YIPL). + +### Successes: + +1. The IATI registry now hosts links to hundreds of files of IATI standard data, covering thousands of aid activities. Over fifty percent of official Overseas Development Assistance will be covered by the IATI Standard now that commitments from major donors have been made to use the standard. +2. Previously, it was difficult even for the governments of recipient countries to know how much money the country was getting from external sources. This affected their ability to be able to effectively prioritise budgeting “[U]nderstanding what donors are estimating to give to our country, whatever sector, helps us [...] to avoid what most ministries may do if we are not on top of information, doubledipping. They come to the Ministry of Finance and request for things that are funded by a donor.” (Brookings Institution Proceedings 2010). +3. Proactive transparency: Lots of Charities (and not only those asked to do so by DFID) are looking at publishing IATI data, as the existence of the standard encourages more organisations to open up what they are doing. The drivers here may in part be around accountability, but also link to motivations of showing donors/public what the charities does, and finding opportunities for collaboration between organisations. +4. The standard has made it worthwhile to build tools that make it easier to publish data in the correct format: such as [openaidregister](http://www.openaidregister.org/) and [Aidstream](http://aidstream.com/public/). +5. The existence of this standard has sparked interest in mapping aid onto other types of financial data such as budgets. When budgets also conform to a standard, such as COFOG, tools can be built to do this work automatically. ‘Interfacing aid information with budgets needs to be tackled at the country level to make information relevant for a specific recipient country, and at donor headquarters level to facilitate the interface at country level. Country aid management systems need to be enhanced to deliver such functionality’ (Mills and Moon 2010). + +### Challenges + +1. There are still simple technical issues - not using the right XML field names, or the wrong format for dates. Initially this was a challenge, but the technical team have been developing tools to help assess data, and working with data publishers on data quality. This can't just be an automated process, as there are differences in data across the world which need human assessment. +2. It is still difficult to cross-reference with other published figures such as budgets and expenditure (particularly when they do not balance). +3. Adoption: While there have been many early uptakers, more work is needed to ensure that other donors and recipients are encouraged to implement the standard. + +### What is Needed to Address These Issues + +1. Quality-assurance tools to automatically check to see that a standard is being used well. Without these, downstream users of the data suffer technical problems. +2. Grass-roots level trainers and community building: Working co-operatively with data publishers as far upstream as possible to check that the content of data is reliable. If this only happens when data comes to be used, lots of people have to duplicate the effort to make the data consistent. +3. Data-management and merging systems: Publishers generating data in the IATI format might have data for different parts of the IATI Standard in different systems: e.g. project financials in one system, results in another, and mapping in another. If IATI data is generated from the projects system only, other content is missing from the IATI form. Having tools to allow merging of data, or allowing anyone to apply annotations to the data, would be valuable. +4. Auxiliary information: A lot of information is locked up in the documents that IATI files can refer to (projects can have related documents), better linkage between them would be helpful. +5. Tools for entity/concept extraction: Entity extraction tools allow the users to identify which individuals, organisations, projects etc. (i.e. entities) are present in a document, allowing them to understand relationships and which documents are related to which transactions and projects. If systems for visualising and presenting data throw the links to these documents away early on, then users’ transparency needs are not well served. +6. The promotion of a standard for transactional level spending data could be useful. This would enable tools to be built which could automatically map aid data onto other budgetary information. + +AidInfo Labs documents plans, prototypes and products which make use of and build on IATI data. These can be found on their website, under the inspiration tab. + +### Learning Points: + +Getting organizations to share their aid data in a sustainable manner and to institutionalize the sharing of aid data through IATI takes time. IATI is already tremendously successful, but it is a long-term project and initial estimates number of donors publishing with IATI could be seen as somewhat optimistic. + +Adapting organisational processes to cope with change management, and system adaptation to enable organisations to start publishing fluid, planning stage information, was difficult for organisations that were accustomed to more statistical reporting (DAC and CRS) and required more work than was initially expected. + +

    Thanks for input from Tim Davies, Michael Roberts and Mark Brough

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter4-2.md b/examples/openspending/content/resources/gift/chapter4-2.md new file mode 100644 index 00000000..fe7bae2a --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter4-2.md @@ -0,0 +1,44 @@ +--- +title: XBRL +--- + +## International Financial Reporting Standards: XBRL + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    + +The eXtensible Business Reporting Language (XBRL) is a data standard for financial reporting, most commonly used to cover accounting information. The goal of XBRL is to efficiently create and disseminate data in a standardized, XML-based format that lends itself to [analytics](http://www.sec.gov/spotlight/xbrl/what-is-idata.shtml). XBRL makes few assumptions about the semantics of the data expressed in it, instead standardizing a framework in which specific reporting types, such as US GAAP or IFRS can be expressed as [taxonomies](http://www.xbrl.org/FRTApproved), each with a specific set of tags to express the reporting elements which are necessary to comply with the standard. A growing number of such taxonomies are developed and maintained by the [XBRL community](http://www.xbrl.org/FRTAcknowledged). + +XBRL has enjoyed widespread adoption as a reporting language for business accounting data across a number of countries, the most notable being the United States, where both the FDIC (2005) and the SEC (2009) have adopted the standard - by 2013 even international companies will have to file their reports to the SEC in XBRL/IRFS. There is little doubt that within a few years, XBRL will be the global lingua franca for accounting information. Going further, the [XBRL General Ledger](http://www.xbrl.org/GLTaxonomy) aims to make the standard usable within companies, storing detailed transactional and other operational information. + +Of course, this comprehensive commitment raises this question: If XBRL is accepted by governments worldwide for receiving data, shouldn’t it also be used to report their own finances? While there is some discussion on the use of XBRL for fiscal information, no taxonomies exist for budget information or transactional government spending. One concern regarding the use of XBRL as a reporting standard for government information may be its complexity: the XML schema requires the implementation of extensive standards to be interpreted. This would run contrary to the goal of transparency by raising the barriers of entry to the analysis of government financial releases. The use of common taxonomies, on the other hand, is a very desirable property, as could also be seen in the IATI example. + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter4-intro.md b/examples/openspending/content/resources/gift/chapter4-intro.md new file mode 100644 index 00000000..d139e789 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter4-intro.md @@ -0,0 +1,20 @@ +--- +title: Standards for Fiscal Data - towards an international framework +--- + +## Chapter 4 - Standards for Fiscal Data: Towards an international framework + +Standards + +Raw data is the fuel of many of the tools featured in this report. In later sections, we demonstrate that the poor quality of the available data and the fact that it is often in non-machine readable formats (e.g. PDF) can contribute considerably to the workload involved in developing these tools, or even render them entirely infeasible.’ + +One of the recommendations of this report is to work towards a standard for transaction level spending data to enable greater reuse and comparison of, as well as confidence in, the data from a user perspective. + +To demonstrate the impact that such an internationally recognised standard can have, the [International Aid Transparency Initiative standard is showcased here](../chapter4-1). We look at: + +1. how a successful standard can be built with multi-stakeholder cooperation and +2. the possibilities this approach has opened up for aid transparency + +We also briefly examine existing accounting standards which may inform how such a standard might look, focusing particularly on the [eXtensible Business Reporting Language](../chapter4-2). + +
    Next »
    \ No newline at end of file diff --git a/examples/openspending/content/resources/gift/chapter5-1.md b/examples/openspending/content/resources/gift/chapter5-1.md new file mode 100644 index 00000000..bebafdd7 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter5-1.md @@ -0,0 +1,83 @@ +--- +title: Government of Sierra Leone GoSL - Online Repository for Ministry of Mines and Mineral Resources +--- + +## Government of Sierra Leone GoSL: Online Repository for Ministry of Mines and Mineral Resources + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageSierra Leone
    URLhttp://sierraleone.revenuesystems.org/
    CostsThe mineral rights administrative system and the online repository have been implemented over a period of two years, costing approximately EUR 400,000 (including training, support and equipment for an administrative system)
    Project executed byRevenue Development Foundation (RDF)
    + +GoSL + +### Background + +A system that allows governments to publish all mining licences and associated revenues directly from their administrative system. The system can also present other types of revenue data such as forestry concessions, fishing rights, property and land rights. + +The system aims to increase government credibility and investor confidence in licenses issued by the government, and thereby increase foreign investments. The system currently has over 500 users, 65% of whom are mining companies and investors looking to verify government issued licenses and companies' good standing. + +Data comes directly from government systems and supports the Extractive Industries Transparency Initiative (EITI) by making audits of government receipts easier and accessible to the public. This generates a close accountability cycle between the public and government, as compared to EITI audits, where audits of government-received payments usually only takes place 2-3 years after they are made. + +The Sierra Leone government is in control of when and what to publish to the public and/or stakeholders, but typically publish updated data bi-monthly. The system contains all mining rights and license payments from 1 January 2010, as well as all exporters licenses and export taxes from 2008. + +Developers behind the site have attempted to make integration with existing government database management systems as simple as possible. Data is imported using an XML schema, which most database systems are compatible with. + +The online portal software can be provided for free and hosted by the Revenue Development Foundation to government institutions. However, it requires an administrative system to be in place from which the data can be extracted. + +### Challenges and Lessons + +Most government institutions in poor countries require extensive support to improve their administrative processes and systems in order to allow public scrutiny of their data. The GoSL Online Repository is the result of RDF support to the Ministry of Mines and Mineral Resources in Sierra Leone which has been given since 2009. RDF has seconded staff to their Mining Cadastre Office throughout the period, in order to change the administrative processes and improve data management. The Ministry use RDF's Minerals Cadastre Administration System (MCAS) to manage the mining licenses they issue, and the data is published to the online portal directly from this system. + +While the initial expectations were that civil society organisations would be prime users of the online data, experience has shown that mining companies and investors are the primary users, alongside law enforcement agencies. + +### Scalability and Future plans + +The system has been designed to be as flexible and scalable as possible. The backend can import a range of government license-and revenue-related datasets. +The GoSL Online Repository was launched by the Government of Sierra Leone in January 2012. Similar systems are currently being deployed for Liberia and Gambia. Work is planned to start in Mali. + +

    Thanks to Aasmund Andersen, RDF

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter5-2.md b/examples/openspending/content/resources/gift/chapter5-2.md new file mode 100644 index 00000000..0390238d --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter5-2.md @@ -0,0 +1,109 @@ +--- +title: Revenues from Oil and Mining - Revenue Watch +--- + +## Revenues from Oil and Mining: Revenue Watch + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageGlobal-level
    ScopeNational governments
    URLhttp://www.revenuewatch.org/
    Users/AudiencesCivil society, national governments, international institutions, researchers
    + +RWI + +### Background + +New technologies and new trends in transparency have unlocked wide new streams of government data about oil and mineral wealth. But access to data does not guarantee access to knowledge. Like the raw materials that get converted to energy, disclosed information often needs refinement and a functioning infrastructure of expertise, analysis and advocacy before it becomes "combustible" fuel for change. + +Revenue Watch tools are aimed at helping to "follow the money" and make sure that wealth in the ground translates into development and economic growth above the ground. RWI and partners work to make the management of resource wealth more responsible, more transparent, and easier to understand. New technology is an increasingly vital part of this effort. + +Revenue Watch has spent more than seven years promoting access to information as an indispensable tool for improving resource management for the public good. Over the last two years, they have been working to define and pilot effective uses of web technology for the display, analysis and dissemination of oil, gas and mining governance data. + +### An Overview of Tools and the Role of Technology + +The [EITI report analysis tool](http://data.revenuewatch.org/eiti) is a tool built by Revenue Watch based on information extracted from over 50 national reports from the Extractive Industries Transparency Initiative (EITI), a voluntary standard in which governments, civil society and companies work together to report payments and government receipts from oil, gas and mining. RWI’s tool presents per-country analysis using RWI's own indicators of report quality, features for comparison of different country results, and easy tools for sharing and downloading data. + +To demonstrate the significance of disclosure rules for international oil and mining companies, RWI launched an [interactive calculator](http://data.revenuewatch.org/listings) illustrating the number and value of extractive sector companies listed on the largest global stock exchanges. This research and the streamlined presentation by value, by exchange and by sector, allows non-experts to explore the data themselves, and better understand the high stakes in new US and proposed EU laws requiring companies to come clean about what they are paying to governments of resource-rich nations. + +A core principle of Revenue Watch’s technical work is that gaps in strategy or in familiarity with online modalities are the main obstacles to better uses of tech for transparency, as opposed to gaps in technology investment. Revenue Watch believe that by developing not only digital tools, but also replicable approaches to data distribution, usability and user training, they can set a higher standard for the use of oil, gas and mining information by governments, companies, advocates, journalists and citizens. + +In addition to these two data tools launched during 2011, Revenue Watch also recently created a simple, highly visual interactive for comparing oil company profits with oil producing economies, at the self-explanatory web address [OilvsWorld.com](http://www.revenuewatch.org/issues/dodd-frank/oilvsworld) (as well as OilvsAfrica.com, OilvsEurope.com, OilvsAsia.com, OilvsAmericas.com). + +The [Revenue Watch Index](http://www.revenuewatch.org/rwindex), a pioneering measure of oil and mining disclosure practices in more than 40 countries, was launched in 2010 as a way to break down and assess transparency practices. The index, created in partnership with Transparency International, is an assessment and comparison of information published by governments about revenues, oil savings funds, sovereign wealth funds and state-owned enterprises, contract terms and other key data. It is an important tool for elected officials, policy makers, civil society and the media when seeking increased public disclosure about natural resource management and greater government accountability. The 2012 index will feature a much richer set of underlying data which will be available for breakdown and comparison by region, country and indicator. This new version of the index is supported by Global Integrity's research platform [Indaba](http://www.getindaba.org). + +Revenue Watch is also currently collaborating with the Open Knowledge Foundation (OKFN) to release an interactive database of original research compiling fiscal rules and royalty practices for mining in African countries. + +### Specific User Skills and Usability + +To maximize not just usability, but “use” of the tools, Revenue Watch worked on design for less-web-savvy users, working in lower-bandwidth settings, possibly on older computers. To maximize dissemination across social networks, where personal connections ensure information shared is more relevant and thus more persuasive, Revenue Watch paid particular attention to tools for sharing the data, including custom links to share user-created views (building on the examples of the World Bank and Google, among others). And, given the reality that the most effective advocacy and analysis often happens "offline", particular emphasis was given to making custom views and full data sets easier to download or print for later use - particularly in the case of the EITI data. + +As new tech approaches are adopted and integrated further across RWI programming, their tools and advocacy will prioritize easier adoption of technology and smarter integration of tech by all actors in the natural resource governance sector. + +### Successes + +RWI's analysis of EITI reports and the EITI data tool represent the first ever compilation of the numerous country level reports in one place and extraction of the report data into a sortable online database. Taken together, these tools helped to demonstrate the value of EITI reporting and provide the first one-stop destination for this data for users in the field, including, for instance, officials at the International Monetary Fund, who decided to use the tool as a resource rather than building their own EITI repository. EITI Chair Clare Short has often cited RWI's research and report tool since it debuted at EITI's 2011 global conference. Most importantly, the RWI tool has served as a model for EITI undertakes its own plan to harvest report data and offer the data online. + +### Issues with the Data and Standardisation + +One challenge within the EITI data has been issues of comparability and standardisation. RWI had to carry out additional calculations and make certain decisions in order to enable more meaningful comparisons to be drawn from the data. This is inevitable given the nature of EITI reporting itself. + +One challenge especially illustrative of the difficulties of creating data sets for oil, gas and mining governance came whilst RWI were carrying out research into oil and mining companies on the largest international stock exchanges. A widespread practice by companies of listing the same securities on multiple exchanges made it difficult to accurately calculate the cumulative market capitalization of oil and mining companies across multiple exchanges. Even the data provider most commonly used as a source for securities data (Bloomberg) turned out to have data that was still full of "cross-listings" of the same securities. "Cleaning" the data to approach a more accurate estimate of market capitalization (a core statistic for the tool's presentation of oil and mining sector value) was a labor-intensive task that, while replicable, was not one that automation could solve due to the level of "noise" in the data. This basic statistic - How much is the sector worth on a given exchange? - is itself very difficult to determine. This is taken as a further sign that better practices in the industry are need in order to reach effective fiscal transparency. + +A good dataset does not guarantee a good database or tool. Most importantly, even a working tool that offers simple access and downloadability of the data does not guarantee usability or "understandability" of the data. This step is the one most often short-changed in the open data activity cycle. Data is not always actionable simply because it is available. A simple interface designed according to user-centric principles, with tools that give the end user power to use the data according to their needs, and which has context and guidance to show what the data means instead of just what it ”is”, are all fundamental elements of the broader "accessibility" that make data leverageable for greater knowledge and further advocacy. + +### Project resources + +The two main data sites mentioned above required several steps to build: data collection, data analysis, tool creation and interface design. A total of only 4-5 people were involved across all phases, but it is important to note that the skill sets required for each of these elements are quite different. Good data collection does not guarantee good data modeling for comparability (or deduplication, as in the stock calculator example). + +### User comments: + +> “Of the sources you mention the only one I have regularly used is Revenue Watch's Stock Exchange Calculator. It's been very useful for comms purposes (as a place to source a fact/figure to support my [argument](http://publishwhatyoupay.org/newsroom/blog/why-dodd-frank-1504-won%E2%80%99t-undermine-competitiveness))”. + + +>“The Oil vs the World tool is good but could be even more useful with a different set of indicators (I'm not sure about the comparability (even if for the sake of the effect of contrast) between GDP and oil company profit. More effective could be to see revenues generated per country for their natural resources vs population living on less than 1 dollar; or oil company profit vs cost of compliance etc...)”. + +

    Thanks to Jed Miller, RWI

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter5-intro.md b/examples/openspending/content/resources/gift/chapter5-intro.md new file mode 100644 index 00000000..dab052dc --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter5-intro.md @@ -0,0 +1,17 @@ +--- +title: Case Studies - Where Does the Money Come From? +--- + +## Chapter 5 - Case Studies : Where Does the Money Come From? + +Revenue + +In order to assess how much money is available to spend, a government must know how much money it receives. Governments receive revenues in many ways, ranging from taxation to income from natural resources. This section examines tools for transparency regarding revenue and the processes around extractive industries. Transparency around taxation policies is covered in the section on participatory budgeting. + + +Case studies include: + +1. [Government of Sierra Leone GoSL](../chapter5-1): Online Repository for Ministry of Mines and Mineral Resources +2. [Revenues from Oil and Mining](../chapter5-2): Revenue Watch + +
    Next »
    \ No newline at end of file diff --git a/examples/openspending/content/resources/gift/chapter6-1.md b/examples/openspending/content/resources/gift/chapter6-1.md new file mode 100644 index 00000000..095087c0 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-1.md @@ -0,0 +1,85 @@ +--- +Title: ClearSpending from the Sunlight Foundation +--- + +## ClearSpending from the Sunlight Foundation + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageUSA
    ScopeFederal level
    URLhttp://sunlightfoundation.com/clearspending/
    Users/AudiencesCivil society, national governments, international institutions, researchers
    + + +Clearspending + +### Background + +Clearspending analyzes the data quality of the grants data in USASpending.gov, the cornerstone of President Obama's transparency platform even before he took office. He co-sponsored the bill that created it whilst he was a Senator, and oversaw part of its execution in the executive branch. Clearspending addresses three metrics: timeliness, completeness, and consistency with other government estimates. Over $1.3 trillion failed on at least one of these metrics. Users of Clearspending can view topline data quality measurements by agency, or drill down to a program level to view the performance of specific grant and loan programs. Essentially, Clearspending is an oversight tool that examines the compliance of each agency and program with grant and loan reporting requirements. The site also provides substantial background information on the evolution of the reporting systems that power it (some are decades old) and the overall picture of how spending is reported in different areas of the federal government. The original methodology of Clearspending stems from a Government Accountability Office report on the same topic, but the sample-based methodology has been expanded with an automated program to examine all transactions instead of only a sample percentage. + +### Target users + +The targeted users are anybody who uses USASpending.gov to get spending data on grant and loan programs. This usually includes, but is not limited to, journalists, academics, policy analysts, concerned citizens, and congressional staffers. For anyone looking at a particular grant or loan program, Clearspending can tell you how much of the program's actual spending data you can expect to find in USASpending.gov. You can also see how on time a particular program or agency is in reporting their spending. Many agencies and programs report far past the 30 day statutory requirement, so their spending can vary widely depending on when you view the data, even after the given fiscal year has passed. As an auxiliary use, Clearspending is also designed to be used by anyone who would want to investigate which agencies or programs are not fulfilling their legal mandate to publish spending information online in a complete and timely manner. + +### Successes + +The project saw considerable public success as an oversight tool. The House of Representatives Committee on Oversight and Government Reform convened two hearings where the Executive Director of the Sunlight Foundation, Ellen Miller, was invited to testify regarding Clearspending analysis, one on the subject of achieving fiscal transparency and one on evaluating open government initiatives. Agency CTOs with particularly poor examples of data quality were also invited to testify. Follow-up questions and reports from the Committee resulted in the correction of some of the most notable errors. + +Most of the errors that were pointed out were related to non-reporting programs. Specifically, the Department of Agriculture had chosen not to report for several programs because they had misinterpreted the guidance. They later wrote to the Oversight Committee explaining this, and pledging to begin reporting the spending. Also, after demoing the beta version of Clearspending to the department of Health and Human services and showing them the gap in medicare/medicaid reporting that Clearspending highlighted, they began back reporting Medicare data. + +### Challenges + +The main challenges of the site stemmed from the complicated picture of the federal spending systems themselves. Sketching a broad view of how the system is designed to work and at what points it is failing proved difficult for most casual users, and even some very knowledgeable users. Federal spending data is nearly impossible to simplify without losing a good deal of information that is valuable to experts on the subject. + +### Data Problems + +The major problems with the data were programs that, for whatever reason, chose not to report their spending. Additionally, many programs reported their spending long after the 30 day window in which they were supposed to, and sometimes even an entire fiscal year later. Another counterintuitive problem was the issue of over reporting. Confusion on the reporting guidance led to student loans being reported in the $6-7 trillion range, instead of the actual $60-80 billion range. + +### Future Plans + +The Sunlight Foundation re-released Clearspending with new data in Fall of 2011, and there are plans to update the analysis each year with new data. More ambitious aims include expanding the analysis to contracting data, which is a more difficult task, due to the lack of the kind of comparison data that was present for grants. Several FOIA requests for information regarding contracts data were in progress at the time of the report. + +

    Thanks to Kaitlin Lee

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter6-2.md b/examples/openspending/content/resources/gift/chapter6-2.md new file mode 100644 index 00000000..5325c6b1 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-2.md @@ -0,0 +1,43 @@ +--- +title: The Media and Fiscal Data - How do we get fiscal data into the public eye? + +--- + +## The Media and Fiscal Data: How do we get fiscal data into the public eye? + +![Europe's Missing Millions from the Bureau of Investigative Journalism](http://farm9.staticflickr.com/8157/7295526880_5f5db8d268_o.png) + + +One of the key promises of open data is that it can be used by journalists to get a clear, evidence-based picture of government action, in particular by using fiscal information to track budgetary priorities, contractual commitments and patterns of spending. As part of the OpenSpending project, the Open Knowledge Foundation is working closely with journalists to develop tools which aim to make financial data released by governments accessible and usable in a journalistic context. We are regularly approached by journalists asking for advice on where to find information on a particular aspect of government spending. In this section, we highlight a couple of our experiences from building these tools, and document some anecdotes from other organisations who have had varying degrees of success with reaching out to the media. + +As a part of this project (Spending Stories), we identified the following challenges: + +1. Journalists are often not used to working with raw data, and don’t consider it a necessary foundation for their reporting. Sourcing stories from raw information is still a relatively new idea. +2. They also often do not possess the necessary literacy and technical skill to analyse the data and present their findings. This includes technical skill, but also statistical and design capacity. +3. Analyzing and understanding data is a time-intensive process, even with the necessary skills. Fitting this into a short-lived news cycle is hard, so data journalism is mostly used in longer-term, investigative projects. This is reflected in a statement by the Budapest Institute in Hungary: “One of the conclusions of the workshop with journalists and the following consultations with media representatives was that our initial idea [of building a tool that would be useful for journalists] turns to be a false hope. [...] We have learned that the Hungarian media is prone to get ready-made analytics and reports rather than to perform investigative projects on its own.” +4. The data released by governments is often incomplete or not up-to-date, making it harder to argue a story based on the available information. Very often, released databases cannot be used for investigative purposes without the addition of FOI-requested, more specific pieces of information. In an article in the [Guardian Datablog](http://www.guardian.co.uk/society/datablog/2011/mar/18/public-finance-data-store) which followed the release of the UK Local Council spending data, Lisa Evans explained why, despite the impressive show of proactive transparency from the UK government, many barriers still prevent journalists from using the data to hold the government to account. The issues she identified include concerns about the completeness of the data; for example, the inability to see money which is allocated in [Private Finance Initiatives](http://www.guardian.co.uk/politics/pfi) (Public-Private Partnerships). + +A couple of observations on Spending Stories as a tool: + +1. We decided early on that we would not confine ourselves to providing journalists with technological solutions, but would also offer training sessions in the use of data and relevant tools. Feedback from participants indicates that this method training was useful. It has also helped us to gauge the level of skill which we can expect users of our technological solutions to have. +2. We started under the assumption that our tool would be aimed primarily at journalists, but have found it is more commonly used by advocacy groups, who have more time to dedicate to research. These advocacy groups then contact journalists with the story in order to gain publicity for the topic once an outline of the story has been researched and the bulk of the data work has been completed. + +When we explored the specific information- and tool-requirements for journalists who wanted to work with spending data, the following needs were recurrent: + +1. Heuristics for story-finding: This could include the automated calculation of statistical measures such as percentiles, scatterplots, standard deviations - but also very specific analysis such as measuring the size of firms in relation to the industry (Herfindahl- Hirschman Index). +2. Time-series visualisations of total spend to date by quarter: This would allow journalists to detect problematic practices such as those highlighted by Vivek Ramkumar’s report: “if the majority of expenses are incurred in the last quarter of the year, this could indicate that the agency was keen to spend money even if it meant wasting it so that it could apply for the next installment of funding in the subsequent year.” +3. Notification services for journalists: When an interesting dataset is published, journalists would get a notification directly to their inbox. When data is open, the determining factor is not who gets exclusive data, but who has the skills to quickly analyse and interpret the data. +4. Search options - how to check whether the datasets (or similar datasets) have been published anywhere)? Some journalists also asked for tools to identify hotspots - to let them see what the most common searches were so that they could get an idea of the topics readers were most interested in and target their stories around those. +5. Ability to call up structured data around the budgeting process is very important: Such information is required in order to follow debates and to see how different MPs voted, as well as to follow amendments, when appropriations were changed, and who was responsible for making the changes. This also includes easy reference to e.g. laws which affect certain spending patterns laws which affect certain spending patterns, ideally presented in a handbook or short glossary format. +6. Linking the numbers to people and organisations: Numbers can be offputting - journalists need to put faces to them in order to make them accessible. Other requests in this vein were a personality tracker - people are particularly interested in following particular high profile names, so it would be interesting if you could follow particular people. For example, having information which lets journalists say ‘here are all the laws undersigned by a particular politician- are there any patterns’? +7. Disaggregated data or more information to understand provenance of aggregates: What is the source of the data? What is the formula via which the end result is reached? + +### Read more: + +* +* +* +* [Sunlight Foundation: Reporting group](http://reporting.sunlightfoundation.com/SLRG/) + +
    Next »
    + diff --git a/examples/openspending/content/resources/gift/chapter6-3.md b/examples/openspending/content/resources/gift/chapter6-3.md new file mode 100644 index 00000000..0c7c22d1 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-3.md @@ -0,0 +1,135 @@ +--- +title: Deciding where the money goes - Participatory Budgeting +--- + +## Deciding where the money goes: Participatory Budgeting + +### e-participatory budgeting + +Participatory budgeting (PB), put simply, gives citizens direct input into how government money is spent, for example via budget consultations or in-person meetings. PB occurs in over 1500 cities over the world, and there is a [Google map](http://bit.ly/IS8Skf), which shows (marked with red pointers) locations which have favoured a technical solution, referred to here as ‘e-PB’. + +For government officials interested in e-PB, the Participatory Budgeting Unit have produced a useful paper including a list of considerations which governments [should take into account](http://bit.ly/PB-role_of_technology). +The collective knowledge of the [Participatory Budgeting Google Group](http://groups.google.com/group/participatorybudgeting?pli=1_) is another useful resource for anyone seeking advice on implementing PB, using technical methods or otherwise. Many members of the group develop e-PB systems, and the group has built a [census](http://bit.ly/PB-software-census) of existing technical solutions. + +The census shows that there are a large number of competing software solutions (both proprietary and open-source) but no real consensus on a particular approach to implementation. When it comes to building a software tool for a process as complex as participatory budgeting, perhaps one size does not fit all. In what follows, we have sought to determine activities in the PB process for which the projects in the census may be able to provide solutions. + +At the end of this section, we present three in-depth case studies of different technical solutions. + +### PB Activities targeted by technology + +The following bullet points highlight key activities in the PB cycle and outcomes which some of the case studies highlighted in this section seek to target: + +1. **Outreach**: Encouraging the ‘hard to reach’ to participate, typically the disenfranchised, the busy and the apathetic. Targeted outreach is also important, being able to contact people who will be most affected by proposed policies or projects. +2. **Following projects and collecting feedback**: The ‘hard to reach’ could equally be defined as finance officers and decision makers in government who have little in-person contact with their citizens. Technology can offer them a way to connect with their citizens. +3. **Publicising progress**: To demonstrate clearly what effects the participatory voting system is having, e.g. which projects were funded in previous years by using this system, how many participants etc. +4. **Aiding deliberation and facilitating debate**: Structuring arguments and collecting ideas e.g. for potential new projects. +5. **Remote participation**: Some people who are unable or unwilling to participate in person may be willing to do so online or via mobile. +6. **Reducing costs**: Commonly incurred costs during a PB process are outreach, planning, running the meetings and collecting the feedback. +7. **Reducing workload**: Running the process can put a lot of stress on staff particularly around planning in the run up to and follow-up from meetings. In offline participatory budgeting, substantial amounts of outreach work and tasks such as translation for minority groups are often taken on by volunteers. +8. **Raising additional funds**: Governments are frequently strapped for the money to implement projects. If additional funds are required to get a project off the ground, some projects highlighted in this section turn to citizens or businesses to make in-kind or cash contributions to get the project off the ground. +9. **Increasing budget literacy**: Simulators allow participants to explore how certain spending/revenue choices impact the budget and gain an appreciation for how much state projects cost. This allows citizens to explore trade-offs between different options. + +### Possible solutions + +This section highlights existing initiatives which take advantage of the opportunities highlighted in the section ‘PB Activities targeted by technology’ . + + +#### Outreach + +**Using technology to invite people to take part in offline voting** + +* e.g. geo-targeted SMS (See the Mobile-Enhanced PB case study, where, when asked how they had heard about the participatory budgeting meeting, over 50% of respondents cited the SMS invites they received as the reason), +* automatic phone-calls where the voice of the mayor encourages people to come out to vote (good for illiterate populations, and hearing the voice of an authority figure may help convince people to participate), + +**Combining online and offline approaches** + +1. e.g. putting e-voting stations outside churches and other places where large numbers of people congregate. + + +#### Following projects and providing feedback + +Gol App + +**‘Problem-solving approaches’** + +* e.g. [Fix My Street](http://www.fixmystreet.com/) / [SeeClickFix](http://makehoustongreat.com/seeclickfix/) or Lichfield council’s [Fix My Tweet](http://www.fixmytweet.com/) - where citizens identify tangible problems e.g. potholes on the street and submit a request via their mobile / via Twitter to their local authority to have them fixed. + +**Following the progress of projects** + +* e.g. [The Gol Mobile App](http://itunes.apple.com/us/app/porto-alegre/id479880903?mt=8) for Porto Allegre is a good example of an application which allows users to track the progress of a request and submit progress reports via their smartphone. + +#### Publicising progress + +* Publicising information on the outcomes of previous rounds of participatory budgeting and what has become of the projects. +* [Bürgerhaushalt Lichtenberg](http://bit.ly/yE7xGa) publishes updates on projects suggested by citizens, including implementation stage, relevant authorities overseeing the work and other comments regarding the projects. + +#### Deliberation + +**Online deliberation:** + +* Many projects allow citizens to submit their own project ideas. e.g.: The UK’s YourLocalBudget project, [‘Budget Ballot’](http://budgetballot.com/) +* Allowing direct feedback into laws already in draft stages. e.g. [Adhocracy](https://adhocracy.de/), which allows users to cut, add to, reword and restructure proposed bills +* Structuring argument. Example: The [MIT deliberatorium](http://cci.mit.edu/klein/deliberatorium.html), which requires users to ‘map’ their argument e.g. pro-contra a point made by another, follow-on questions, ideas for solutions to previously raised problems. + +**Offline deliberation, aided by tools.** + +* Example: ‘America Speaks 21st century town-meetings’ are in-person meetings which use computers on every table to serve as “electronic flipcharts” to record general table agreements; table agreements are instantly transmitted to a “Theme Team”, which reads who read all the entries to identify the strongest themes. These overarching themes are displayed and quickly presented to all the participants; individuals use their individual voting keypads to vote on what they believe are the most important priorities. + +#### Remote participation: + +* online voting and surveys (numerous examples in census. See also the Citizen Budget case study). + +#### Cost reduction: + +* Outreach e.g. advertising meetings on social media +* Email - usually cheaper than paper outreach. +* Even SMS can be a relatively cost effective option. In the Dominican Republic, the World Bank negotiated with phone companies a rate of around $0.01 per message. A single callout on radio cost around $200 to advertise the meeting. Nonetheless, text messages appeared to be a more effective option in terms of the number of people motivated to join the meeting (see statistics in the Mobile-Enhance Participatory Budgeting case study). + +#### Reduction of workload + +* Systems which help to manage and direct volunteer efforts: e.g. collaborative editors, (Google Docs), Microtask management (Tasket), Translation ([Amara](http://www.universalsubtitles.org/en/), [Transifex](https://www.transifex.net/)). + +#### Raising additional funds + +* Very few [Kickstarter](http://www.kickstarter.com/)/[Pledgebank](http://www.pledgebank.com/)- style applications exist for publicly funded projects, however this approach has been incredibly successful in crowdsourcing support for non-governmental projects via the internet. One of the few attempts to do something similar is: [Leih Deiner Stadt Geld](https://www.leihdeinerstadtgeld.de/) in Germany, this works by encouraging people to invest in their city. + +#### Increasing Budget Literacy + +* Budget simulators: The Estonian site [Meieraha.eu](http://meieraha.eu/) allows people to visualise and explore the effect of different revenue and expenditure policies e.g. raising taxes. +* Personal Impact Calculators: One aspect which very few of the budget simulators dealt with was personal impact, where users are shown what the impact on them personally would be, e.g the effect that increasing spending above budget levels would have on the amount of tax that they as an individual would pay. Large numbers can be disorienting for the average citizen who may find it difficult to relate the numbers to more familiar measurements such as household budgets and their daily salary. An example attempting to solve this problem is [Budget Allocator](http://demo.budgetallocator.com/), which explains to users the impact of their choices: “Although you have balanced your budget, based on your current selections, we may still have to increase rates by 6.4%. Typically a 1% rates increase equates to 12 cents a week.” +* Research is currently in progress at the Ash Centre in Harvard into forcing people to watch an explanatory video before voting. This will obviously not work in all situations (e.g. places with bandwidth issues), however it does have the benefit over text that it ensures that information cannot be ‘skipped’. + +### Concerns about using technology in PB + +The concerns below were voiced about using technology in PB by those both inside and outside governments: + +* “Possibilities for gaming the system” (e.g. with bad voting systems). +* ‘Herding effects’ from special interest groups, which could be difficult to monitor if participation is remote. +* Using technology as the ‘easy option’ and not dedicating enough time to face-to-face interaction. +* For both citizens and governments: Not having the right skills to be able to deal with such a system. +* Creating additional work for government officials by allowing torrents of unstructured, unmoderated and often duplicate comments to flood in, all expecting answers. +* Information cascades in online voting systems: Due to the volume of responses, participants often do not have time to review all of the proposed solutions, so if only the most recent comments and suggestions appear at the top, or only top ranked, they may gravitate towards selecting these. +* Who participates? Particularly, where remote participation is concerned: Ensuring that only relevant constituents are participating. +* Finding the right balance between a simple and usable application and an oversimplified solution. + +### e-PB: What can policymakers do to help projects such as these succeed? + +This section has dealt mainly with tools for direct participation, as opposed to indirect participation, where citizens choose their preferred policies via the proxy of an elected representative. In the latter case, there are also ways in which governments can help to promote informed decision-making in electing their preferred candidate. For example, by releasing structured information on voting records of a given candidate. In countries such as Kenya, where individual politicians hold sway over a large discretionary fund (Constituency Development Funds), having access to information on an individual’s previous financial behaviour can be crucial to selecting the correct representative. See [this post](http://openspending.org/blog/2012/01/30/hakuna-my-data-nbo-data-bootcamp.html) on ‘Auto-generated campaign speeches’ for one suggestion of a project to compare candidates based on their financial track-record. + +Many of the projects in this section are civil society led and are listed here to provide inspiration for some of the ways in which technology may be used to promote greater citizen participation around budgeting issues. However, these efforts can only go so far without support from governments. The following section highlights a couple of ways in which governments could promote such projects as these. + +#### Promoting public participation in the annual budget cycle, and in the design and delivery of public services and public investment projects + +1. Publish machine-readable schedules of key events in the budget cycle e.g. publication of key documents, consultation periods etc. +2. Make sure teams within governments have the resources to deal with potentially large volumes of feedback in order not to disappoint those in their constituencies by not being able to take suggestions into account. +3. Make sure technology does not become a substitute for face-to-face contact. Many important parts of the PB process happen offline. Technology is a useful complement to offline channels of communication and participation, but is not a substitute. + +#### Promoting participation around legislation concerning budgeting issues + +1. [Examine structured data formats for drafting legislation and tabling amendments](http://www.publicwhip.org.uk/project/liaison.php). +2. Be open and transparent about [how legislation is drafted](http://www.ictparliament.org/xmltraining_brussels2012). Progress was made recently when the European Parliament (EP) decided to [open-source At4am](http://bit.ly/KhOr13), software that helps staff at the EP write and table amendments. Such moves help those developing tools for deliberation and collaboration around ideas outside government understand workflows and adapt their tools to work with existing government workflows. +3. Note that while the European Parliament plans to release the At4am software, [it has not currently announced a plan to make the data from its own copy available](http://www.asktheeu.org/en/request/source_code_and_data_for_at4am_s_2), and doing so would represent a substantial increase in transparency around legislators’ activities. +4. Structured information on who in a department is responsible for [particular changes](http://pudo.org/2011/07/22/kleine-charta-der-beteiligten.html). It is unproductive to attempt to talk to an entire department about a proposed project or change, who are the people who can actually make a difference? + +
    Next »
    + diff --git a/examples/openspending/content/resources/gift/chapter6-4.md b/examples/openspending/content/resources/gift/chapter6-4.md new file mode 100644 index 00000000..40f196db --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-4.md @@ -0,0 +1,122 @@ +--- +title: Mini atlases for participatory budgeting - Solo Kota Kita (Solo, Our City) +--- + +## Mini atlases for participatory budgeting: Solo Kota Kita (English: Solo, Our City) + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageIndonesia
    ScopeLocal government
    URL
    Users/AudiencesDonors, recipient country governments, citizens, NGOs
    + +![SKK Image](http://farm8.staticflickr.com/7231/7295628522_e2dce3fc25_o.png) + +### Background + +In 2001, Indonesia underwent ambitious decentralization programs. The "big bang" reforms devolved political, administrative and economic power to smaller administrative units, and local actors took over management of municipal infrastructure and systems. Against this backdrop, the national as well as local governments developed and supported new planning processes that sought to increase community participation in government decision-making. + +Participatory budgeting (known in Indonesia as Musrenbang) was piloted by a group of NGOs in Solo for the first time in 2001. In 2005 the national government mandated that every region and city in Indonesia implement an annual Musrenbang process. As part of the Musrenbang process, residents in a city or region meet to prioritise and propose short-term improvements they'd like local government to implement in their neighbourhoods. + +While participatory processes such as Musrenbang are in place, in many cases local authorities and communities lack capacity or tools to maximize the collaborative decision-making opportunities these processes offer. For example, priorities set during Musrenbang meetings don't always correspond to the most urgent requirements of a given neighbourhood. SKK’s first project sought to tackle this issue in Solo by increasing awareness of urban issues to strengthen residents’ voices and their capacity to prioritise community issues during the Musrenbang process. + +### Project Description + +The tool, ‘the Mini-Atlas’, presents a map and thematic information about each neighbourhood in Solo. Each atlas includes a map of community facilities (responding to local feedback that often the participatory budgeting meetings happen without a map of the neighbourhood) and thematic data regarding education, water, sanitation, housing, poverty, and health, highlighting key assets and issues for residents to discuss. + +After completing the mini-atlases, SKK, in coordination with the city government and Musrenbang facilitators in each neighbourhood post large copies of the mini-atlases in neighbourhood public spaces. Residents can also print-off this information themselves and use the maps whenever they like. + +Data Collection and Analysis Methodology +Solo city government had never collected the information included in the mini-atlases, let alone aggregated it through GIS and disseminating it more widely, but the administrative organisation of all cities across Indonesia, including Solo, facilitates the collection of this data. + +Each neighbourhood in Indonesian cities is formed of administrative units called RTs (roughly equivalent to a US Census Block). A “block captain” keeps track of how many people live on the block, how many poor families, etc. SKK put together a team of community facilitators that liaised with the 2,700 “block captains” in Solo to collect the information to populate the mini atlases. As such, each mini-atlas provides a highly detailed profile of each neighbourhood. + +### Resources + +*Data collection*: 6 months in total. + +*Working with data*: GIS mapping and analysis, design of the mini-atlases, website with searchable information from the atlases. (local GIS consultants and a California-based firm to produce SKK website) - 6 months total over a six-month period. + +Note that following the data collection, SKK found differences in the collected data compared to official statistics e.g. the official population estimate is lower than SKK’s count. Rather than SKK having over-estimated the population, it seems more likely that some people were not counted in the official census, perhaps due to dissemination problems, illiteracy, or inadequate follow-up procedures. While such data discrepancies exist, they are generally not critical. + +### Scalability + +The experience with the data collection emphasises that this methodology is replicable. As discussed in ‘Follow up projects’ SKK will attempt to replicate the initiative in Solo and Makassar, adding an innovative SMS data collection approach. + +### On the Ground Implementation: Using the Data + +SKK trained all Solo Musrenbang facilitators in the use of the mini-atlas during the 2010-2011 Musrenbang process. Further, SKK staff collaborated closely with Musrenbang facilitators in 10 neighbourhoods to encourage and observe the utilization of the tool during that Musrenbang cycle. + +During the 2011-2012 Musrenbang cycle, SKK staff conducted refresher training for Musrenbang facilitators. Unlike the previous rounds, in four of the 10 neighbourhoods, SKK staff worked with neighbourhood leaders and facilitators to develop a second-generation mini atlas. The latter, known as MA+, sought to amend/ augment the original neighbourhood mini atlas with information that community members noted would enhance the effectiveness of the tool. For example, in one neighbourhood, feedback included adding information about small businesses within the mini atlas. + +### Follow-up Projects + +SKK has begun a Musrenbang budget analysis initiative with the aim of increasing transparency and developing new planning tools related to the outputs of the Musrenbang process in Solo. To achieve this aim, SKK is reviewing and comparing the lists of proposals made at the RT levels to the respective final city budgets, to determine which projects proposed through Musrenbang were selected and implemented. Ultimately, this information could be utilised to develop maps that visually highlight what type of projects were proposed and where, as well as which were implemented in previous years. + +Additionally, with funding from the Ford Foundation, SKK will begin collecting RT-level information from each neighbourhood utilising an innovative SMS data-capturing methodology. Specifically, instead of community facilitators and “block captains” filling out paper forms to aggregate data, SKK will pilot an approach by which the data collection can be conducted using an SMS based cell-phone application. In addition to conducting this work in Solo, SKK will also implement the project in the city of Makassar. + +### Successes + +1. In some neighbourhoods the mini atlases served as a catalyst for communities to enhance their discussion of issues and prioritisation of Musrenbang funding requests. +2. The Solo government officially mandated and included future funding for the printing and use of the mini atlases as part of future Musrenbang cycles. +3. As part of a city-sponsored design competition, ‘Solo Eco Cultural City’, to create solutions for a sustainable city, student participants downloaded and used the mini atlases to do their planning. +4. Other city and NGO initiatives are utilising information from the mini atlas to help inform their work. For example, a current City-NGO initiative which will conduct a poverty assessment in a number of Solo neighbourhoods is utilising the mini atlases as an information baseline. +5. A critical characteristic of the mini atlases was providing clean, universally recognised graphics that could be understood by community members. +6. The mini-atlas is a tool with relevance beyond Musrenbang. Musrenbang is part of a continuum of planning activities in neighbourhoods that includes RPJMK planning, which is medium-term poverty alleviation planning; PNPM, a World Bank upgrading program, and others. + +### Challenges + +1. The willingness and ability of neighbourhood facilitators to effectively utilise the mini-atlases has a significant impact on whether they are used or serve their purpose. Some neighbourhoods resisted the use of new tools. In other neighbourhoods, facilitators were not as engaged as they should be with the Musrenbang process (for example, some do not show up to community meetings as they should). SKK is currently working on a set of recommendations to tackle this. +2. Feedback during RT and RW (groups of RTs) meetings suggests that in many communities the data presented in the mini atlases is too high level. In other words, the mini atlases do not provide sufficient nuance regarding the blocks since they are presented at the neighbourhood level. This issue was a key catalyst in SKKs 2011-2012 effort to develop a second-generation mini atlas. +3. Given the 18-month Musrenbang budget cycle and a lack of clarity regarding the projects selected and implemented during previous Musrenbang processes, there is a tendency for communities to recycle project proposals yearly. SKK’s ongoing Musrenbang budget analysis project seeks to shine light on this issue. + +### Project Resources + +Key Actors: A project director, 2 community facilitators, an urban designer, a GIS expert, a website developer,10 community facilitators to communicate with and collect data from RTs, student volunteers (local universities) as map makers/drafters. + +Additionally, it is important to note that the project had the support of the Solo mayor. The government worked with SKK to engage the neighbourhood heads to support data collection. + +

    Thanks to Hector Salazar Salame and Michael Haggerty (SKK)

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter6-5.md b/examples/openspending/content/resources/gift/chapter6-5.md new file mode 100644 index 00000000..21a47f38 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-5.md @@ -0,0 +1,114 @@ +--- +title: Balance your City's Budget - Citizen Budget, Open North +--- + +## Balance your City’s Budget : Citizen Budget, Open North + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageCanada
    ScopeLocal government
    URLhttp://citizenbudget.com/
    Users/AudiencesAll levels of government
    + +Citizen Budget + +### Background + +Citizen Budget is a project of Open North, a Canadian non-profit that creates websites to increase government transparency and promote citizen participation. Open North sells Citizen Budget to municipalities as licensed software or as a service. + +Citizen Budget is an online budget simulator that focuses on a local government's controllable operating expenses. The tool challenges citizens to re-balance the budget by increasing or decreasing the amounts allocated to municipal activities, local services and revenue sources. The municipality defines the list of activities and services and their associated amounts. + +Once the online consultation is over, Open North prepares a report for policymakers at the municipality, to allow them to better incorporate citizens' priorities in their budgetary decisions. This report evaluates how representative the participants are of the population, analyzes the data, and identifies trends, which it reinforces with data visualizations. +The goal is to consult citizens on their budget priorities, while educating them about the local budget and raising awareness of the difficult choices and trade-offs in balancing a budget. Other potential customers include police and fire authorities, school districts and labor unions. + +Citizen Budget's first client was the borough of Plateau Mont-Royal in the City of Montreal, regarding their 2012 operating budget. Open North is now looking for municipalities who would like to consult citizens on their 2013 operating budget. This project can be tailored to allow citizens to submit input on printed forms or using their mobile devices. + +### How it works + +Unlike many of the applications in the census, which asked users to vote on abstract priorities e.g. increase health spending by X million (voting in amounts which, without specialist knowledge, do not mean an awful lot to the average citizen who (generally relate more easily to e.g. the scale of their household budget), citizen budget allows users to vote on concrete and tangible outcomes, which are aligned with decisions that councillors might have to make themselves (build more of these, agree to close libraries on Sundays to save money.) Citizens have to produce a balanced budget, so are forced to trade-off priorities against one-another. The voting system is incredibly simple and allows the user to understand the trade-offs and impacts of their given choice in context. + +### How long did the project take to implement? + +The project is constantly improving and so work is ongoing. The current iteration of the project took 10 days for the technical implementation, with another 10 days for data collection, meetings with government, etc. + +### Successes + +Online applications can reach a larger and more diverse group of citizens and can offer more personalized information than traditional offline methods. Unlike council meetings or face-to-face consultations, citizens can participate online anytime, from anywhere, using their favourite Internet device. + +They can take the time to express themselves carefully without having to wait to be given the floor. In short, online solutions reduce many barriers to participation. + +This project has so far been run in one city district, the borough of Plateau Mont-Royal in the City of Montreal. Out of an adult population of 89,000, 3,160 visited the budget simulator and 363 submitted balanced budgets that expressed their budget priorities. The local government used the major trends in residents' budget priorities to inform their budget decisions - principally, to allocate more money to make the borough greener and cleaner. + +Residents expressed their appreciation for the consultation and gained a greater appreciation for the difficult choices and trade-offs that elected officials face when balancing a budget. + +Many citizens from outside the borough expressed interest in having a similar initiative in their territory. + +### Challenges + +1. The project did not have government staff dedicated to the project's promotion, and so did not reach the greatest number of participants. +2. The government did not effectively communicate to citizens the way in which it used the input from the online consultation. +3. The major challenge is for the government to determine the actual cost of individual activities and services. Budgets do not tend to be organized per activity, and so a fair bit of work has to go into coming up with these numbers. + +#### Do users require any special skills to use the platform? +The project requires citizens to have a basic level of competency in the following: Internet use, reading skills, budget literacy, and familiarity with municipal activities and local services. + +#### What skills are required to implement the project? + +The project requires a web developer and a designer to create the application and a statistician/analyst to prepare the report on the consultation for the government. Additional non-technical staff are needed to educate and interact with governments, to manage the other team members, to plan the promotion of the consultation, etc. + +### Other comments + +1. Consultations cost money. With many local governments facing significant budget constraints, many find it difficult to justify (to themselves and to residents) the cost of a consultation. + +2. Politics. Some governments may worry about the consultation being a negative experience, especially if residents are already not showing support. They may worry that residents will support a policy or initiative that goes against the government’s promises, platform or beliefs. Finally, a particular consultation method may be seen as a vestige of a previous government, and therefore something to get rid of. + +3. Many local governments would want an offline consultation in addition to any online consultation, e.g. to get input from harder-to-reach demographics. Even if Open North, as a service provider, makes the online consultation as easy-to-run as possible, some local governments will not use it due to the planning, preparation, operating and follow-up costs of an offline, face-to-face consultation. + +4. It is difficult to get a local government to run a consultation year after year. Most will perform it as a one-off initiative. It is considered more in terms of its publicity, marketing or communications value, rather than its democratic value. + +Such systems have the potential to better inform and educate citizens about decisions and processes. By sharing decisions with citizens, they can become more efficient and effective in the delivery of services. Governments provide much of the information for these initiatives. However, without government buy-in, these initiatives would not have the power to effect change. + +

    Thanks to James McKinney (Open North)

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter6-6.md b/examples/openspending/content/resources/gift/chapter6-6.md new file mode 100644 index 00000000..75db365f --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-6.md @@ -0,0 +1,119 @@ +--- +title: Mobile-Enhanced Participatory Budgeting - the World Bank +--- + +## Mobile-Enhanced Participatory Budgeting : the World Bank + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageDemocratic Republic of Congo (also been done in Dominican Republic) +Mobile Voting has also taken place without the Bank in: Ipatinga Brazil, La Plata (Argentina), Belo Horizonte (Brazil)
    ScopeLocal government
    Users/AudiencesIndividual citizens
    Source DocumentThis section contains excerpts from Estefan and Weber’s short article
    + +### Background + +After years of conflict in the Democratic Republic of Congo, many are unable to access the most basic of public services. While many citizens in the province don’t have access to water or electricity in their homes, they do have mobile phones. The World Bank Institute’s ICT4Gov program has introduced[m] mobile technology to enhance participatory budgeting processes to help local authorities decide on the allocation of available resources according to citizens’ priorities. In a community-chest style process, the local government devotes a percentage of the local investment budget to the project deemed most important by the citizens. + +The World Bank played the role of facilitator in a number of spheres for the project - they tailored software, negotiated with cellphone operators and built capacity to demonstrate to local populations how the technology works. In each use-case, the Bank provided the local community with a portfolio of options that technology could offer and allowed the community to decide for themselves which options to deploy. The Bank was also instrumental in other processes such as drafting the text messages which were used to contact constituents and building capacity in understanding the budget. In the case of the Democratic Republic of Congo, the Bank played a role in bringing local governments, civil society, universities etc. together to work out how to implement these projects. + +### The Role of Technology + +ICT4Gov is using mobile phones for four purposes: + +* To invite citizens to participate offline in participatory budgeting assemblies through geo-targeted SMS messages. These messages, reaching all the phones receiving signal from a particular tower, announce the date, time and location of the assemblies. + +Besides the blanket approach, the Bank is also experimenting with manually collecting phone numbers (this was done in the Dominican Republic). When collecting numbers, it is also possible to collect basic information about the person in question, such as their gender, and the neighbourhood they live in, so that targeted messages can be sent out (e.g. low attendance rates from women can be counteracted by sending invites just to women.) So far, the Bank have sent around 2000 messages this way compared to 300,000 through the blanket approach. + +* Mobile phones were also used for voting (DRC), allowing the citizens to send a text to identify which of the priorities they would like to see addressed in their community. This is being trialled in a controlled environment at the in-person meetings. This ensures that citizens are present throughout the discussions and so are able to make informed decisions. During meetings, the participants rank their priorities on a scale of 1-4. This makes meetings more efficient as counting is done electronically and feedback can be monitored. In the Dominican Republic participants had the option to vote remotely. + +* As an announcement service to communities to broadcast the the result of the vote, making the process more transparent and inclusive. + +* For feedback on the projects that were chosen. Through text messages, citizens are able to offer feedback and monitor the projects. + +![Perception DR](http://farm8.staticflickr.com/7245/7272472242_28b8506119.jpg) +

    +Voters in the Dominican Republic were asked how they felt about having the option to vote remotely via SMS. +

    + +### Project Resources + +1. Knowledge of participatory design and community operation +2. Development knowledge in mobile telephony sector +3. Outreach skills +4. Cost of SMS: In the DRC one million messages were negotiated at 10,000 USD + +### Timeframe + +1. Training on PB: 8 cities - 2 months +2. Development: 2 months +3. Running a session: 2-4 hours + + +### Scalability + +There are 1500 cities around the world where participatory budgeting already takes place, and this approach could be considered as an option in them. Using mobiles could be used to encourage citizens is an approach that could be used in other contexts to promote citizen participation, not just PB. In some countries, such as Estonia, mobile voting has already been used in municipal elections and could be extended to other contexts. + +###Successes + +1. When the participants were asked what motivated them to take part in the meeting, the majority of respondents mentioned that the SMS had been the strongest deciding factor. +2. Since the beginning of the program, as the Provincial government sees an increase in the capacity of local government to allocate resources, communities involved have already seen an increase in transfer of funds from the Provincial to the local level. +3. The preliminary results of an external evaluation suggest that an increase of tax collection at the local level has been associated with the implementation of participatory budgeting. This is based upon testimonials offered by officials and numbers collected with the municipalities. However, given the high number of potential exogenous factors that could have contributed to this result, further assessment should be done before claiming a causal relationship. For the first time, communities such as Ibanda have gone from not having any investment budget to having 40% of their budget devoted to investments. In 2011, the Ministry of the Budget started the process of institutionalisation of Participatory Budgeting in the Province. +4. The increase in the transfer of funds from the provincial to the local level has benefited communities, which now have more resources to deliver public services to the poor. For instance, the process has made it possible to begin repairing 54 classrooms and a bridge in Luhindja, to create a health centre and repair the sewage system in Bagira, and to build a water fountain as well as toilets in local markets in Ibanda. + +![Find out 1](http://farm9.staticflickr.com/8164/7272472560_e4338cbdd7.jpg) +![Find out 2](http://farm9.staticflickr.com/8017/7272472430_f6ee268408.jpg) +

    +Participants in the Dominican Republic were asked how they had found out about the meeting. +

    + +### Challenges + +1. Collecting numbers requires a lot of resources (Dominican Republic) +2. With some bulk message providers, some messages did not deliver until many hours later, some recipients received messages in the middle of the night! +3. More work needs to be done to help people monitor the execution of the projects. People are engaged in the process and feel empowered when they vote, but there is little follow-up and people could begin to become disillusioned with the project if they do not see results. + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter6-intro.md b/examples/openspending/content/resources/gift/chapter6-intro.md new file mode 100644 index 00000000..bf065023 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter6-intro.md @@ -0,0 +1,25 @@ +--- +title: Chapter 6 - Introduction +--- + +## Chapter 6 - Case Studies: Where Does the Money Go? + +Spending + +The following case studies all feature organisations who re-use the fiscal data released by governments about where money is spent. They seek to promote better specialist and public understanding of the data and increase participation in the budgeting process. + +There are two main sections within this ../chapter: + +### Presenting Data to the Public + +1. [Grading Government Spending](../chapter6-1): Clearspending, Sunlight Foundation +2. [The Media and Financial Data](../chapter6-2): how do we get this info into the public eye? + +### Deciding where the money goes: Participatory Budgeting + +1. [Introduction](../chapter6-3) +2. [Mini atlases for participatory budgeting](../chapter6-4): Solo Kota Kita, Indonesia +3. [Citizen Budget](../chapter6-5) - Open North, Canada +4. [Mobile-Enhanced Participatory Budgeting](../chapter6-6) - the World Bank + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter7-1.md b/examples/openspending/content/resources/gift/chapter7-1.md new file mode 100644 index 00000000..921f5753 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter7-1.md @@ -0,0 +1,105 @@ +--- +title: Introductions +--- + +## From Fractions to Millions: Challenging corruption using mobile phones + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageIndia
    ScopeLocal government
    URLbit.ly/fractions-to-millions (Vivek Srinivasan's article). All quotes from Vivek Srinivasan’s article.
    StagePilot Project
    Users/AudiencesRural poor and beneficiaries of local government programs
    + + + +### Background + +In recent years, India has been one of the leaders in FOI legislation, where in many aspects the Right to Information Act was seen as more progressive and powerful than legislation in other [developed countries](http://www.guardian.co.uk/society/2012/apr/10/india-freedom-of-information?newsfeed=true). The possibility of digitising public finance records has lead to a significant decrease in the per unit cost of accessing information. "Mazdoor Kisan Shakti Sangathan (MKSS) and other NGOs involved in India’s right to information movement realised that merely having access to government records was not enough given the sheer complexity of the records and people’s ability to understand them." + +Social audits in India typically involve comparing the official records of government projects (e.g., cash books, muster rolls, measurement books, supply lists) and validating whether these projects actually existed in reality. This information verification takes place through door-to-door surveys, where actual records are compared against the locals' testimonies - where discrepancies indicate acts of corruption. "The power of this process is evident by the fact that even though very few convictions happen on the basis of the findings of a social audit, data shows that the levels of corruption have reduced appreciably in places where audits have been organized regularly." + +The method of the social audit involves accessing the official record and verifying it with the person who should have received the goods or services. Yet understanding the particularities of the project requires training and expertise, and the actual survey requires considerable time and resources to conduct. Mobile phones could be used as a technological solution to social auditing. "Through basic mobile phone SMS technology, official records on basic individual entitlements such as pensions, subsidized food grain, and maternity entitlements could be delivered to individuals via monthly text messages." + +“The project has now received the commitment of officials in the states of Bihar and Andhra Pradesh in India and a team that we helped organize is just starting to build the basic technology to store and disseminate public records. We will start with select programs such as the National Rural Employment Guarantee Act in India and expand the coverage over the next few years.” + +### The Role of Technology + +The Indian Census showed that around 50% of Indians have mobile phones even in rural communities, whereas access to the internet is limited, which highlights the potential of using mobile technology to reach out to households without Internet access. "The main benefit of using SMS technology is that it requires very limited skill, knowledge, or effort from the user. And, it eliminates costly and time-consuming in-person surveys and audits. By using this technology, official information can be disseminated on a regular basis, unlike in the current model where social audits are done sporadically." + +### Scalability: + +A planned elaborate survey would compare villages where the technology has been implemented to villages where it has not in order to measure the effect of mobile technology on combating corruption. The existing legal framework and the low cost of making public finance information available mean that more citizens and other organised groups will be able to combat corruption. + +“At a later stage of the project, it may be possible to add other features that will enable the victims of corruption to take action using mobile phones. For example, the SMS could include the phone number of responsible officials, or an NGO could collect the complaints and initiate action on their behalf.” + +### Successes + +1. Using SMS technology will facilitate more widespread and more frequent social audits, which are instrumental in tackling corruption’ +2. “[it] arms individuals with precise information – something they never had before — that officials cannot argue or ignore.” +3. The resource could be used by people with limited skills or access to the Internet + +### Limitations + +"Like any technology tool, this of course has its limitations. One of the critical functions of social auditing in India has been its role in mobilizing the general public. The process of gathering people together face-to-face in a public meeting creates a collective energy, which can motivate people to fight corruption." + +### Issues with the Data + +Much of the data is in printed format and should be digitised in order to conduct the project. + +## Project Resources + +Estimated project cost: 400,000 USD (including an elaborate survey to measure corruption) + +

    Thanks to Vivek Srinivasan

    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter7-2.md b/examples/openspending/content/resources/gift/chapter7-2.md new file mode 100644 index 00000000..e66a94d6 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter7-2.md @@ -0,0 +1,18 @@ +--- +title: Introductions +--- + +## What Can Governments Do to Help Social Auditors? + +There are considerably fewer projects in this section than in the participatory budgeting section. The reasons for this are numerous, but projects often suffer from lack of timely and granular data from governments and the feedback mechanisms to enable citizens to make their voices heard. Promoting participation in government audits should not be perceived as a threat, but rather an opportunity: auditors and civil servants cannot be everywhere, but citizens can, and can provide feedback on issues as and when they perceive them, allowing the government to hone in on problem areas when they need to. A few suggestions: + +1. Collaborate: Support and partner with civil society projects which have already developed auditing tools and use the generated feedback for decision-making and public administration oversight. +2. Implement Standards: Implement accepted data standards and formats in order to ensure the financial data can be categorised and connected back to specific entities and projects. +3. Allow Anonymous Feedback: Mechanisms by which citizens can anonymously submit information regarding conflict of interests regarding MPs, allowing for the disentanglement of the politicians’ interests, where personal benefits might influence the direction of public policy and public procurement: e.g websites such as [Inspector de Intereses in Chile](http://www.inspectordeintereses.cl/), encouraging mobile feedback, or employing official hotlines such as those used by Supreme Audit Institutions (SAIs). +4. Organise Structured Feedback Mechanisms: Beside providing the framework for auditing tools, governments should ensure that the feedback is assessed and acted upon in a structured, timely and efficient manner. This makes it clear who is responsible for assessment and responding to feedback. +5. Provide Online Training and Support: Training classes and guidance materials about the audit processes for other governments, public officials, civil society and interested citizens could be also made available online with webinars, shared presentations and open training tools. +6. Ensure Collaborative and Participatory Process: Establish spaces for cooperation between the SAIs and the civil society organisations as they have mutually complementary roles and resources (Nino 2010). +7. Crowdsource Problem Areas: Providing mechanisms for citizens to give feedback on problems as they experience them. Once an area of concern is identified e.g a particular department is frequently flagged up as asking for bribes when issuing licences, the government would know to investigate further. +8. Lead a Transparent Process End to End: Clear communication and documentation about how the auditing information is used and acted upon. + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter7-intro.md b/examples/openspending/content/resources/gift/chapter7-intro.md new file mode 100644 index 00000000..2bc87131 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter7-intro.md @@ -0,0 +1,16 @@ +--- +title: Chapter 7 - Introduction +--- + +## Chapter 7 - Case Studies: The Invisible Money + +Invisible Money + +Here we examine projects which track institutional corruption and malpractice. Social auditing, in the sense used here, is when non-governmental organisations or collections of citizens ‘audit’ government projects. There are a wide variety of organisations who engage in this important work around the world. As well as the examples listed below, the appendix provides further examples of technology which has been used to tackle problems similar to the issues facing organisations conducting social audits (which include violence against auditors, outreach and spreading the message and text-message based reports of service delivery). + +### Social Audit + +1. [From Fractions to Millions](../chapter7-1): Challenging corruption using mobile phones +2. [What Can Governments Do to Help Social Auditors?](../chapter7-2) + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter8-intro.md b/examples/openspending/content/resources/gift/chapter8-intro.md new file mode 100644 index 00000000..a66db194 --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter8-intro.md @@ -0,0 +1,120 @@ +--- +title: Chapter 8 - Putting the Parts Together +--- + +## Chapter 8 - Putting the Parts Together - Getting off-budget on-budget OpenSpending & Publish What You Fund + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fiscal ScopeProject AimsTechnology
    + + + + + + + + + + + + + + + + + +
    Geographical CoverageUganda (could be re-used in other countries)
    Scope central government and international
    URLhttp://openspending.org (OpenSpending site) http://bitly.com/uganda-example +
    Users/AudiencesDonors, recipient country governments, citizens, NGOs.
    + +OS + PWYF + +### Background +Aid flows often do not pass through a recipient government’s conventional budget mechanisms. When this happens, recipient governments themselves may not have the complete overview of where aid money goes and how donor priorities align with their own. This information is vital for governments and aid donors to be able to make the best use of scarce resources. + +Normally this overview is not available – leading to waste, overlap and inefficiency. The lack of comparable information means aid donors and recipient country governments can’t work together to coordinate their efforts; it decreases developing country governments’ ownership and undermines the potential for good governance and planning. Donors and governments need to know what others are doing - and crucially, what others are planning on doing - if they are to make sure that these resources are used most effectively. Otherwise, some sectors and areas will not receive enough funding, while others may have too many donors involved. + +This project was an effort to combine two key types of fiscal data - revenues from aid and spending information - and present them together in an informative way through an interactive visualisation. + +#### How long did the various stages of your project take to implement? + +Data Collection: The first step was a huge data-collection effort by the Overseas Development Institute (ODI) and Publish What You Fund (PWYF). It took just over 6 months for the initial data collection, cleaning, and report, with a large amount of manual work by many different people. Not only did the ODI have to manually collect all this data on donors’ aid spending, they then had to map it all to Uganda’s budget. For data collection, detailed financial information was provided by the Ministry of Finance and this existing data for each development partner (or donor) was sent to them for verification, correction and completion, in the form of an MS Access database. + +Data Wrangling: 6 months. This data was still not machine-readable and capable of being analysed as it was spread across five different tables. There were other problems as well, e.g. no common currency throughout. PWYF processed the data so that it was in a format suitable for importing into the Open Knowledge Foundation’s OpenSpending software. + +Development: 1 man month. The Open Knowledge Foundation created the BubbleTree visualisation so that it was possible to see multiple dimensions of the data at the same time (you can see both which sectors the money is going to, and how that is broken down by type of spender - donor, project aid, budget support). + +#### What kind of skills/resources were/are required for the projects? + +Expert knowledge of budget and aid data. Data wrangling capabilities. Development skills. + +### Successes + +1. The data collected in this project was far more comprehensive than the data in the Government of Uganda’s budget. In fact, for the Financial Year in which the report was being conducted (2006/7), donors planned to spend almost double the amount of project-based aid compared with what the Government of Uganda was aware of. +2. The project proved that it was possible to collect all the necessary data to be able to do this type of analysis for an individual country. Standards, such as IATI, make it easier for such approaches to be replicated at scale. +3. The visualisations drew attention to a couple of interesting patterns e.g. very occasionally aid money showed up in defence spending, big chunks of money going to disaster management and the north. It also made it possible to establish and compare how donors are (or not) aligning to the policy priorities of the Ugandan government. + +### Challenges + +The project required considerable human resources to clean and collect the data. If data had been published in a consistent and machine readable format, this would have been considerably easier. + +The feedback below comes from people we asked to review the platform: + +1. When we tried to solicit for feedback and encourage journalists to use the visualisation in their reports, they still asked us ‘where are the stories?’. The visualisations gave a high level overview, useful to assess overall priorities and aid distribution, but more work with capacity building to help journalists understand and work with fiscal data is required to help them find the stories. +2. Another reason journalists were reluctant to cover the story was that the data was not up-to-date enough. The most recent data available was from 2006 and the visualisation was completed in 2011. +3. Further work needed on explanatory texts - what exactly can you do with the visualisation and what are the known limits? +4. Visualisation inevitably implies a level of editorial judgement. We received a comment that if Uganda receives general budget support, all the bubbles should show a sliver of aid (since general budget support funds the overall budget, not specific sectors). In the current version, many sectors do not show a budget support component. The decision to show this at the top level only, and not in each of the sectors, was taken because by that point (i.e. at the sectoral level) it is then considered part of the Government of Uganda’s revenues. + +### Scalability + +OpenSpending is an international platform which allows anyone to upload and visualise government financial data. The database already holds many international, national and sub-national datasets and the software can easily be translated into other languages. OpenSpending is open source and open to contributions on any level from the community; contributing data or code and translation are the most common activities. + +A similar approach could be taken in other countries, and work currently underway on the standardisation of aid-sector codes may make this easier. In addition, possibilities of mapping IATI compliant aid data onto COFOG compliant budgets are being researched, although more granular standards may be required in order to accurately represent the data[1]. If standards were widely adopted, this approach could be replicated. + +To extend the project further and make it even more useful, the following are needed: + +1. Aid information (but crucially, it needs to be timely, detailed and comparable): Donors representing over 30% of global Official Development Assistance already publish their data as of 18/11/2011, but more work will be needed to encourage the other 70% to do so. +2. Budgets in a machine-readable format: some governments have already begun publishing these (e.g. Kenyan open data initiative); many others have budgets available online in PDF (including Uganda, Sierra Leone, for 2010). +3. Mapping from aid to budgets: work is currently underway in this area. + +### Where next? + +Possible further areas for exploration and development of the platform: + +1. Seeing budget in perspective of the legislative process, although this is a) difficult and b) only part of the story, as a lot of the most interesting changes happen when the Budget has been agreed and moves to the Executive. +2. Heuristics: On a basic level, show average, maximum, minimum aid donations to a sector. Variance: flagging differences above/below a certain amount +3. Comparisons, e.g. Spending relative to other districts. Ability to see the context of your current view: (what filters have you selected?). +4. Feedback and comment features, ability to annotate data points as well as collections / facets of the data e.g. to show absence of data. + +

    Thanks for input from Mark Brough

    + +
    + [1]COFOG, which was designed specifically to describe the activities government undertakes, is an appropriate starting point to examine alignment between recipient budget classifications and existing international aid classifications. COFOG represents country sector and administrative classifications fairly well at aggregated levels, but at the lower levels tends not to disaggregate the functions of government in the same ways or to the same degree that many governments do‘ (Moon and Mills, 2010). +
    + +
    Next »
    diff --git a/examples/openspending/content/resources/gift/chapter9-intro.md b/examples/openspending/content/resources/gift/chapter9-intro.md new file mode 100644 index 00000000..4022539b --- /dev/null +++ b/examples/openspending/content/resources/gift/chapter9-intro.md @@ -0,0 +1,88 @@ +--- +title: Final Observations and Review +--- + + +## Final Observations and Review + + +While this report demonstrates possible ways in which technology can contribute to fiscal transparency it should also be clear that technology is not a magic potion that will automatically resolve all barriers to fiscal transparency. + + +The authors of this report were tasked with highlighting ‘cutting edge’ technology for fiscal transparency. While we have attempted to do so and to show some of the very best and most sophisticated work in this sphere , we would like to emphasise that simple solutions often yield the best results. + + +In delivering on fiscal transparency, a combination of online and offline, technical and non-technical approaches will be required. In some cases, technology will be neither necessary nor appropriate: e.g. in participatory budgeting it will be crucial to engage citizens offline if the process is to be effective. In other cases, prioritising ICT over offline methods can lead to exclusion for poor and marginalised groups who have little access to the internet or low literacy levels. + + +Nevertheless, the examples highlighted here do show that technology can be a very effective complement to existing processes in enhancing fiscal processes and transparency. These tools can work on numerous levels simultaneously, reducing barriers to participation and reaching out to groups who would otherwise not have the chance to participate. + + +We have already highlighted specific recommendations in individual sections, and here we would like to pull together some of the most important. The report demonstrates that many actors need to be involved in order to promote fiscal transparency: + + +Below are numerous recommendations related to the role of all parties including governments, civil society organisations and multi-stakeholder initiatives such as GIFT. + + +### Governments: + +* Promote the use of simple, machine-readable formats which can significantly increase the usefulness of data +* Make sure that all fiscal data is released under a proper open license (i.e. one that conforms to [the Open Definition](http://opendefinition.org/)) +* Support and partner with civil society projects to assess whether existing, open-source tools will serve the purpose desired, before looking to create new ones +* Organise structured feedback mechanisms and make sure teams within governments have the resources to deal with potentially large volumes of feedback +* Crowdsource Problem Areas: Providing mechanisms for citizens to give feedback on problems as they experience them. Once an area of concern is identified e.g a particular department is frequently flagged up as asking for bribes when issuing licences, the government would know to investigate further. +* Make sure technology does not become a substitute for face-to-face contact. Many important parts of the PB process happen offline. Technology is a useful complement to offline channels of communication and participation, but is not a substitute. +* Examine structured data formats for drafting legislation and tabling amendments + +### Technical Specialists + +* Look into ways to make it easier for governments to make data available from the data management systems they already work with - do they have APIs? can they easily pull out reports from their accounting system (e.g. an ‘export report for press’ option)? + +### Civil Society Organisations & Private Sector + +* Re-publish data they have derived and added value to. +* Work on capacity building and data literacy: Are all necessary skills being fostered to maintain technical solutions? For example, is an intern relied on to load new data or do all members of staff know the standards and processes required to do it? + +### Foundations & Funders + +* Many of the projects highlighted in this report are one-off solutions, tailored to an individual local problem. Funders may like to review existing projects (e.g. those listed in the appendix & in the participatory budgeting software census, and on sites such as the [Civic Commons Marketplace](http://civiccommons.org/apps)), to see whether any existing open-source solutions could be adapted to serve other or more general problems, before initiating another. +* Support platforms (as well as individual instances of a piece of software) which allow solutions to be scaled, promote inter-organisational data-sharing and communities of practice. +* Collect resources for long-term archives of data and documents relating to fiscal activities. + +### GIFT: Recap + +First and foremost, we see GIFT as being in a strong position to foster technical standards and best practices regarding fiscal transparency. It should be appreciated that especially the release of data, by its nature, is generative: it is nearly impossible and undesirable to predict what people will want or be able to build when data is made both available and usable. Such outcomes should be specific to each country, addressing its political and technical landscape. + +Hence, in this report, we have attempted to highlight case studies which are experimental in their nature. We hope that GIFT will be instrumental in supporting further such experimental projects and promoting discourse between governments and civil society in this area. + +#### Promote Open Data as the Raw Fuel for Technology + +* Members of GIFT should themselves ensure - whether they are national governments or international organisations - that they publish their fiscal and other financial information in full and in machine-readable formats. + +* Promote publication of key budget information as machine-readable data. Work with organisations such as the International Budget Partnership to include a related criterion in the Open Budget Survey. Set up online webinars / workshops showing how data is used after its publication. + +* GIFT should initiate the creation of a light-weight, demand-driven standard for the release of structured expenditure information to enable its comparability between countries. + +* Work closely with the Open Government Partnership to promote release of datasets which are key to contextualising financial data, such as economic indicators or procurement data. + +* Promote the release of sub-national data, perhaps via local partners, as well as national level. + +#### Technology Outreach and Idea Sharing + +* Promote experience sharing between governments through workshops and forums in existing social media platforms. Governments need to share the lessons learned in using technology to promote transparency, accountability and participation across different countries. + +* Involve civil society actors who use data and services provided by governments, to review and share ideas about existing and planned initiatives. Such actors could form advisory-panels on particular topics e.g. company data. + +* Create a well-maintained index of existing initiatives both an the demand and supply side, including methods of data release, re-use and presentation, discussion or activism enabled by such information. + +#### Technology Review + +* Review existing projects and and solutions from the point of view of both governments and civil society actors.[Civic Commons Marketplace](http://civiccommons.org/apps) as well as the [appendix](http://bit.ly/TTAPF-projects), but a larger-scale, community effort will be required to keep this up to date. + +* Based on feedback from the above refine and maintain an online, collaborative catalogue of technical and policy options (perhaps as a wiki) which can guide governments about existing solutions and experiences. + +#### Technology Sharing + +* Promote the use of promising projects from the Technology Review stage. Liaise with funding bodies to ensure open-source solutions are available for all. + +
    Next »
    \ No newline at end of file diff --git a/examples/openspending/content/resources/gift/images/citizen.png b/examples/openspending/content/resources/gift/images/citizen.png new file mode 100644 index 00000000..e14c2855 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/citizen.png differ diff --git a/examples/openspending/content/resources/gift/images/data_analysis.png b/examples/openspending/content/resources/gift/images/data_analysis.png new file mode 100644 index 00000000..21ceac8d Binary files /dev/null and b/examples/openspending/content/resources/gift/images/data_analysis.png differ diff --git a/examples/openspending/content/resources/gift/images/decision-maker.png b/examples/openspending/content/resources/gift/images/decision-maker.png new file mode 100644 index 00000000..01e6821d Binary files /dev/null and b/examples/openspending/content/resources/gift/images/decision-maker.png differ diff --git a/examples/openspending/content/resources/gift/images/educate.png b/examples/openspending/content/resources/gift/images/educate.png new file mode 100644 index 00000000..43c2eb66 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/educate.png differ diff --git a/examples/openspending/content/resources/gift/images/invisible_money.png b/examples/openspending/content/resources/gift/images/invisible_money.png new file mode 100644 index 00000000..700e52e7 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/invisible_money.png differ diff --git a/examples/openspending/content/resources/gift/images/logo.png b/examples/openspending/content/resources/gift/images/logo.png new file mode 100644 index 00000000..db5a3879 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/logo.png differ diff --git a/examples/openspending/content/resources/gift/images/logo_small.png b/examples/openspending/content/resources/gift/images/logo_small.png new file mode 100644 index 00000000..ddd535e2 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/logo_small.png differ diff --git a/examples/openspending/content/resources/gift/images/mobile.png b/examples/openspending/content/resources/gift/images/mobile.png new file mode 100644 index 00000000..ad57c936 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/mobile.png differ diff --git a/examples/openspending/content/resources/gift/images/offline.png b/examples/openspending/content/resources/gift/images/offline.png new file mode 100644 index 00000000..e60218c6 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/offline.png differ diff --git a/examples/openspending/content/resources/gift/images/on.jpg b/examples/openspending/content/resources/gift/images/on.jpg new file mode 100644 index 00000000..0f3aec3f Binary files /dev/null and b/examples/openspending/content/resources/gift/images/on.jpg differ diff --git a/examples/openspending/content/resources/gift/images/piechart.png b/examples/openspending/content/resources/gift/images/piechart.png new file mode 100644 index 00000000..6cfe57cc Binary files /dev/null and b/examples/openspending/content/resources/gift/images/piechart.png differ diff --git a/examples/openspending/content/resources/gift/images/radio.png b/examples/openspending/content/resources/gift/images/radio.png new file mode 100644 index 00000000..4d456f70 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/radio.png differ diff --git a/examples/openspending/content/resources/gift/images/revenue.png b/examples/openspending/content/resources/gift/images/revenue.png new file mode 100644 index 00000000..cab065d4 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/revenue.png differ diff --git a/examples/openspending/content/resources/gift/images/social_media.png b/examples/openspending/content/resources/gift/images/social_media.png new file mode 100644 index 00000000..81d7ce04 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/social_media.png differ diff --git a/examples/openspending/content/resources/gift/images/spending.png b/examples/openspending/content/resources/gift/images/spending.png new file mode 100644 index 00000000..ce40b950 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/spending.png differ diff --git a/examples/openspending/content/resources/gift/images/standards.png b/examples/openspending/content/resources/gift/images/standards.png new file mode 100644 index 00000000..3cf9ebda Binary files /dev/null and b/examples/openspending/content/resources/gift/images/standards.png differ diff --git a/examples/openspending/content/resources/gift/images/upload.png b/examples/openspending/content/resources/gift/images/upload.png new file mode 100644 index 00000000..3b6bf627 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/upload.png differ diff --git a/examples/openspending/content/resources/gift/images/user.png b/examples/openspending/content/resources/gift/images/user.png new file mode 100644 index 00000000..fab38329 Binary files /dev/null and b/examples/openspending/content/resources/gift/images/user.png differ diff --git a/examples/openspending/content/resources/gift/images/web.png b/examples/openspending/content/resources/gift/images/web.png new file mode 100644 index 00000000..09747cbe Binary files /dev/null and b/examples/openspending/content/resources/gift/images/web.png differ diff --git a/examples/openspending/content/resources/gift/index.md b/examples/openspending/content/resources/gift/index.md new file mode 100644 index 00000000..4e0174f0 --- /dev/null +++ b/examples/openspending/content/resources/gift/index.md @@ -0,0 +1,27 @@ +--- +title: Global Initiative for Fiscal Transparency Report +--- + +![Cover of TTAPF Report](http://farm9.staticflickr.com/8027/7296195082_0ae9bdb2f0_z.jpg) + +This report was prepared for the Global Initiative for Fiscal +Transparency ([GIFT](http://www.fiscaltransparency.net) with generous +support from the Omidyar Network. The views represented are those of +the authors. + + + + + + +
     
    + + + +
    + +
    + +
    + +{% include gift_nav.html %} diff --git a/examples/openspending/content/resources/gift/pdf/ttapf_report_20120530.pdf b/examples/openspending/content/resources/gift/pdf/ttapf_report_20120530.pdf new file mode 100644 index 00000000..8408e740 Binary files /dev/null and b/examples/openspending/content/resources/gift/pdf/ttapf_report_20120530.pdf differ diff --git a/examples/openspending/content/resources/handbook/.gitignore b/examples/openspending/content/resources/handbook/.gitignore new file mode 100644 index 00000000..3bffeb0a --- /dev/null +++ b/examples/openspending/content/resources/handbook/.gitignore @@ -0,0 +1,3 @@ +old/* +book.zip +book/* diff --git a/examples/openspending/content/resources/handbook/.gitmodules b/examples/openspending/content/resources/handbook/.gitmodules new file mode 100644 index 00000000..e69de29b diff --git a/examples/openspending/content/resources/handbook/NOTES b/examples/openspending/content/resources/handbook/NOTES new file mode 100644 index 00000000..640c2104 --- /dev/null +++ b/examples/openspending/content/resources/handbook/NOTES @@ -0,0 +1,9 @@ + + + +https://github.com/esetera/Objavi/blob/master/templates/templating_template.html + +http://okfn.booktype.pro/export/spending-data-handbook/export + + + diff --git a/examples/openspending/content/resources/handbook/README.md b/examples/openspending/content/resources/handbook/README.md new file mode 100644 index 00000000..6939e0e0 --- /dev/null +++ b/examples/openspending/content/resources/handbook/README.md @@ -0,0 +1,19 @@ +Spending Data Handbook +====================== + + +How to generate and export this book: + +* Go to booki.cc and log in. +* Visit your dashboard and press import book. +* Select "Booktype book" option and use + http://okfn.booktype.pro/spending-data-handbook/ for the URL; then +import. +* Go to the book and publish it. +* Pick up the output from http://objavi.booki.cc/tmp/?C=M;O=D +* Replace all updated files, esp. body.html +* Open in a Chrome with experimental WebKit features (cf + http://bookjs.net/). +* Print :) + + diff --git a/examples/openspending/content/resources/handbook/Spending_Data_Handbook.epub b/examples/openspending/content/resources/handbook/Spending_Data_Handbook.epub new file mode 100755 index 00000000..bb1d7a11 Binary files /dev/null and b/examples/openspending/content/resources/handbook/Spending_Data_Handbook.epub differ diff --git a/examples/openspending/content/resources/handbook/Spending_Data_Handbook.mobi b/examples/openspending/content/resources/handbook/Spending_Data_Handbook.mobi new file mode 100755 index 00000000..293db565 Binary files /dev/null and b/examples/openspending/content/resources/handbook/Spending_Data_Handbook.mobi differ diff --git a/examples/openspending/content/resources/handbook/body-header.html b/examples/openspending/content/resources/handbook/body-header.html new file mode 100644 index 00000000..4bb5402d --- /dev/null +++ b/examples/openspending/content/resources/handbook/body-header.html @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/examples/openspending/content/resources/handbook/body.html b/examples/openspending/content/resources/handbook/body.html new file mode 100644 index 00000000..0bf831e0 --- /dev/null +++ b/examples/openspending/content/resources/handbook/body.html @@ -0,0 +1,467 @@ + + + + + + + + + + + + +
    Introduction
    1. Introduction
    2. Working with others
    3. Helping the government help itself

    1. Introduction

    +

    The Spending Data Handbook is addressed to people and organisations who want to use and understand government budgets and spending data in their work. These groups may include government oversight and accountability groups, issue oriented advocacy organisations, journalists covering the latest round of budget cuts or austerity measures, or citizens who just want to inform themselves on what their government is spending. The truth is that for many countries, it's never been easier to access the underlying data relating to government budgets and spending and the expectations have never been higher for governments to release this information in a timely manner. But once you've got the information, making sense of it can be daunting.

    +

    Irrespective of which kind of user is approaching this data, they all have one thing in common: they need to manipulate, analyse and interpret data in their work. Increasingly, we are bombarded with statistics and figures from governments and political parties and it is the job of these organisations to scrutinise and fact-check them, as well as come up with alternative models and solutions. This requires timely, data driven analysis.

    +

    If these groups think that "topic-driven analysis" can drive positive change within governments, there is another group at whom this book is addressed: a strange race of people who self-identify as "civic hackers" and think that technology can be put to work to make government as easy, accessible and effortless for citizens as the internet. While these groups use data processing tools in their day to day lives, their focus is often on enabling others to act on information, rather than having a particular advocacy aim themselves (though there are a few notable exceptions to this rule).

    +
    DON'T PANIC! This book uses a bit of technical vocabulary and terms that may be unfamiliar or used differently in different circles. Anything in italics such as hacker in this paragraph, can be found in the glossary at the back, it may not mean what you think.
    +

    The range of topics tackled by these groups is so diverse that it would be impossible to address all of the questions and issues in one humble book. However, there are some overarching principles and helpful techniques which apply universally to working with government financial data and particularly, how technology might help to do so. This book aims to highlight areas where civic hackers, citizens, Civil Society Organisations (henceforth CSOs), and civil servants working in the field of government transparency could combine forces to achieve common aims. Like the Open Data Handbook (http://opendatahandbook.org/en/), it will be available as a continually evolving, open, educational resource on the internet.

    +

    What we will cover in this book

    +
    • Collaborating with other organizations to pool resources and strengthen your advocacy effort
      • +
    • If you're just starting out, what data to look for and what to ask for (nay, demand!) from your government
      • +
    • The 'Data Pipeline': Tricks and tips for finding, wrangling and systematically processing your data
      • +
    • Getting ambitious, running a technology project
      • +
    • Presenting your findings to engage the public, media and government
      • +
    • Lists and appendices of technical and non-technical resources
      • +

    The Data Processing Pipeline

    +
    +

    How the book was created

    +

    The book was started at a four day book sprint bringing together organisations from around the world from both a technical and a CSO background. Representatives from the Open Knowledge Foundation (UK), Fundar (Mexico), the Centre for Budget and Governance Accountability (India), the Public Interest Advocacy Centre (Bosnia and Herzegovina), and the Sunlight Foundation (US) contributed to the original document. The Book Sprint was facilitated by Adam Hyde, founder of the Book Sprint method and www.booksprints.net.

    +

    Why are we writing this?

    +

    As people who work with this data, we know better than anyone that a few years of a handful groups spreading awareness of spending data to the general public has not been sufficient for enacting earth-shattering change in the budgetary policies and processes in governments around the world. While it's true there's more spending data available than ever before, these efforts have proven to be superficial in some cases and the budget processes themselves have not become much more transparent or participatory.

    +

    Conversely, the armchair auditors that were expected to emerge from the citizenry never really materialised. The notion of citizen auditors and engaging with citizens on an individual and collective basis still struggles to make headway. Clearly, we need to take solutions to this problem to the next level. More groups need to be more involved than ever before.

    +

    What makes this work so tricky?

    +
    • The data people need is not available to the public (and to CSOs).
      • +
    • Alternatively, the data is not available until after all the important decisions within government have been made.
      • +
    • It's difficult to simplify data as complex as budgets and spending and make it accessible to a variety of audiences.
      • +
    • The discourse on budgets and governance in the country is usually replete with jargon and technicalities.
      • +
    • Even when compelling research findings are presented, there's no sense of urgency on the part of key policy actors. Advocacy with a range of stakeholders becomes important.
      • +
    • There is duplication of efforts. One CSO may invest two weeks of its resources into painstakingly cleaning up and extracting data from a policy document published as a PDF while another will be doing exactly the same thing.
      • +
    • Organizations doing this work often lack any kind of peer review process among similar groups.
      • +
    • There may be skills gaps at many CSOs. Skill and knowledge sharing can help all of them achieve common goals.
      • +

    Contribute to the book

    +

    This book is released under a Creative Commons attribution licence, meaning that anyone is free to use and reuse the material provided that it is attributed to the Spending Data Handbook. You can help by doing one of the following:

    +
    • Translate it and customise it for your region. The examples which we include here are the ones which we know best, however, you may feel that in your area, there are topics or examples which would be more relevant. Take the book, remix it and add your own examples.
      • +
    • Correct it and update it - treat it like a Wikipedia article, a living document. The only way that this book will stay relevant and factually correct is with the help of you and other people who know your stuff.
      • +
    • Be inspired - we are filling the book with lots of examples of visualisation and data so inspire you and people you work with.
      • +

    2. Working with others

    +

    We've outlined some ambitious goals for this book, but you don't have to go it alone! In this section we focus on how to work with other organizations to create powerful advocacy efforts, all driven by hard fiscal data! Every CSO, not just government transparency organizations, can leverage the information in this book. Government spending data touches issues covered by a variety of organizations and journalistic outlets, such as human rights, the environment, poverty, scientific research, education, economic policy, and more. But not all these groups may realize it's importance.

    +

    Why work together?

    +

    A common observation has been that CSOs, journalists, and other groups that work with a lot of data encounter a set of similar problems. The problems typically include lack of technical knowledge, duplication of work, lack of wide dissemination of analysis and results, and creating compelling visual products. A lot of these challenges could be mollified if proper channels of communication existed between these organizations.

    +

    One of the most important ways to reduce these technical barriers is to maintain an active dialogue and a process of sharing the methodologies and underlying data in analysis you produce. You can avoid the mistakes of your predecessors by reading their methodologies and learning about idiosyncrasies in the data. Or you may be able to point out mistakes another organization has made. Similarly, if you document how your data has been processed and analyzed, other organizations can benefit (more about that in a few chapters).

    +

    Creating communication can be as easy as joining or maintaining a mailing list of groups that work on similar topics or leverage similar data in their work products. Most webmail services offer free group or mailing list functions, such as Google Groups or Yahoo Groups. If you maintain one, be responsive to messages and active in recruiting members. Email is an extremely low barrier for starting a conversation between organizations.

    +

    If you've already got data products that you want to share, try disseminating them in a couple of different ways. In addition to the methods outlined above, try preparing primers or manuals for other groups that want to start working with datasets that you may be particularly experienced with. You can also organize workshops, meetups, and webinars to familiarize groups with the data you work with. If grassroots issue groups can internalize your data products and present them in a contextually relevant way to their user base, then both groups have gained something from the partnership.

    +

    Another good way to facilitate knowledge sharing and communication between organizations is by conducting internship programmes for interested individuals and organisations. However it is important to keep in mind the target groups. It's great to interface with similar spending data focused groups working at different levels of government than your own, but identifying groups that work on completely different issue areas is important. These groups may feel incapacitated to enact change in the fiscal policy as it relates to their own issues. It then becomes an important job for spending data groups to identify and offer their support. It's especially important to establish strong links with groups that serve those minority populations who are often marginalized in government spending priorities.

    +

    The kind of collaboration discussed above can result in a much broader and stronger coalition of organizations that can advocate for fiscal transparency and detailed disclosure of spending data at all levels of government.

    3. Helping the government help itself

    +

    We've discussed that it's important for CSOs, journalists, and other groups to work together and it's likely that you won't bump up against too much resistance to this idea. But what about when you get pushback from the government even after you've built a strong coalition to advocate for better access to data? Well, one answer you can give them is "Government darling, you're only hurting yourself".

    +

    Whenever the word 'transparency' is mentioned, the first thought that usually springs to mind is a bright light being shone into dark corners of government offices, trying to expose the dark secrets of corrupt bureaucrats and inefficient expenditure. What is often overlooked is that governments also often stand to benefit from more transparent publishing practices. Not only does proactive disclosure help save on the time and cost associated with increased Freedom of Information requests, but other governments within the same nation can benefit from each other as well.

    +

    Connecting different levels of government

    +

    Money at the federal level is often transferred to local governments in the form of grants, direct payments or formula payments. When sub-national governments have better, more timely information on national budgeting priorities, it allows them to adjust their own budgets to account for cuts or increases in certain programs. Similarly, many local governments have small staffs and rely on revenue estimates and models at the federal level to estimate their own revenue.

    +

    This underscores the need for a consistent, standardized information sharing across governments, as it would allow them to share models, best practices, and software with each other, instead of custom building everything from scratch. Furthermore, it would allow for cross-checks on both the federal side and local side for data auditing purposes.

    +

    The next section highlights a couple of case studies for CSOs to use when trying to convince their government to publish better data.

    +

    Case study

    +

    Transparency to combat lag-times for Governments in British Columbia

    +

    When the province of British Columbia built a data portal (http://www.data.gov.bc.ca/), its motivations were primarily:

    +
    • citizen engagement - they wanted citizens to better understand the workings of government
      • +
    • innovation - they wanted people to build applications and tools using the data
      • +
    • making handovers effective - a large number of the workforce were approaching retirement age, and those in charge wanted to make sure that they handed over the necessary information well in advance
      • +

    Probably one of the less-expected impacts was seeing how civil servants themselves used the portal. In 2012 approximately one third of all the traffic originated from government computers. The technology enabled faster access to relevant data within the government departments, contributing to better collaboration on policies that required fiscal data. There was also about 20% increase in the number of Freedom of Information (FoI) requests, showing that releasing a small amount of data fuelled wider interest in data.

    +

    The importance of constant access

    +

    It is somewhat terrifying to think that a lot of budgeting decisions could made almost entirely in the dark. As soon as one executive budget proposal is finalised and published, work often begins on producing the next one. Within governments, those who have to draw up the next year's plan need access (and quickly) to information, such as actual quarterly expenditures, in order to work out whether a government department is properly resourced from the outset, or is drastically under or over spending.

    +

    Case study

    +

    International Aid Transparency Initiative (IATI)

    +

    Building on around 20 years of previous work, the first version of the International Aid Transparency Initiative (IATI) standard was launched in early 2011 and aid donors around the world started publishing to it. The IATI specification documents data about both aid donors and aid activities, enabling comparison and encouraging good practice in data management. The IATI standard also includes space to attach relevant documents and details of project results, to allow the standard to build context around the data and become an end-to-end solution, tracking projects from inception to execution.

    +

    The IATI standard has seen far greater uptake than any previous effort in terms of organisations electing to publish their data in a standard format. These successes are often attributed to the multi-stakeholder nature of the design process, including both policy makers and technical experts. From 2009 to 2010, IATI consulted with a wide range of stakeholders on the design of the technical standard, alongside a parallel process to secure donor support for publishing their aid information.

    +

    In the past it was common to respond to information shortages by building a new database. But by working with open data principles, IATI allows a more distributed solution - where information can flow between organisations in many different ways, not just into a central database.

    +

    Main user groups

    +
    1. Parliamentarians in developing countries gain a better oversight of the aid resources available. Knowing where to allocate resources in their own budgeting processes is vital to ensure that money is spent in the best way. Sometimes, there is transparency-asymmetry between different parts of government. For example, a treasury may be very willing to open up the information it holds, but departments which benefit strongly from aid donations (e.g. departments of health) may be more reluctant to be transparent about aid revenues, as they will not want to 'lose out' from central government budgeting.
      2. +
    2. Donors: Know where their money is going and whether it is being spent correctly.
      3. +
    3. CSOs and private companies who monitor aid effectiveness
      4. +

    The benefits of this approach have already been demonstrated, with many CSOs and charities choosing to follow the IATI Standard, although it was developed primarily with governments in mind.

    +

    Donors publish aid information as a feed which can be read by many different applications, including those created by other donors, by the open data community, and - importantly - by software providers that are developing country financial systems. By providing aid information in a standard format, many different users can access the data in the way they need to - and developing countries can see the resources, which are supposed to be flowing to them.

    +

    Is there anything like this for spending and budgets?

    +

    At present, no. The way many governments report their financial data is a function of how their budget process works or how they use their accounting software. This may not even be consistent within a government, but vary by department and sub-department.

    +

    Could it be useful to create one? We suspect so. The IATI standard for the first time is enabling people to track the money across country boundaries, something which is almost impossible with other types of expenditure, but crucial for those whose job is oversight of government activities and accountability bodies, such as those who work to prevent international money laundering and corruption.

    Data Literacy
    4. Data-driven advocacy and research
    5. Types of data
    6. Getting and cleaning data
    7. Analysis
    8. Using technology in your work

    4. Data-driven advocacy and research

    +

    We are now in a phase where many governments around the world are proactively publishing documents about what they plan to spend (budgets) and actually spend (spending data). Increasingly, this material is available on the internet, so that anybody can access it at any time. Still, too much of the information is released in the form of 'documents' rather than 'data'. Ideally we need both so that inforamtion can be analyzed, re-used and understood. This chapter is a quick overview of some of the raw inputs required for data-driven advocacy and how it works in practice. 

    +

    What do we mean by machine-readable data?

    +

    When we speak about data, what we usually refer to is the notion of machine-readable (http://en.wikipedia.org/wiki/Machine-readable_data) data. Many of the formats most commonly used for policy papers and long-form reports published by most policy-making institutions are PDF files, Word documents, web pages or closed interactive infographics - do not structure information in a way that lends itself to automated analysis and extraction.

    +

    Such documents are formatted for humans (or printers) to interpret, and it can be hard (and in many cases nearly impossible) for a machine to re-construct the elements in the presentation.

    +

    Other formats, such as Excel and CSV files contain a higher level of structured information. For example, in an Excel file you can mark a number of cells and easily calculate their sum. Even more exotic and useful file formats, such as XML documents, JSON APIs or Shapefiles may not have easy-to-use viewer applications. You can think of them as the glue that connects different systems on the web, so that different databases can work together in a seamless fashion.

    +

    Why do CSOs need it?

    +

    What asking for machine-readable bulk data means for CSOs is simple: you won't have to spend a lot of time manually extracting data from reports into spreadsheets to be able to filter, sort and analyse it - a process which is both time-consuming and can introduce errors.

    +

    What to ask for when asking for data: a checklist

    +

    In the next section 'Getting Data' - we will deal with asking governments for data (or getting it via other means). To set the scene for this and to work out whether your government actually publishes usable data already, have a quick look at the following questions:

    +
    • Is the government's data published in a machine-readable format? E.g. CSV, XML, JSON. While there is nothing wrong with publishing a PDF to support a data release (in fact it is often nice to have a nicely-laid out document to cross reference and sanity-check data) it shouldn't be the only thing published and if you are asking for a policy document, ask for the underlying data in a spreadsheet so you can check the numbers.
      • +
    • Does the government publish a 'data dictionary' to explain the terms used in the dataset? This should include definitions of column headers, explanations of terms and ranges used within the main body of the data, explanations of any changes in terminology which have been introduced since last time the dataset was released
      • +
    • How is the data that is being published actually used internally by governments? Do some sanity checks on the minimum and maximum values of different columns to make sure they fall into the documented ranges and don't seem out of place. Do you see negative values when you don't think you should? Negative values usually mean money owed.
      • +
    • Is the structure of the data the same across years? If not is there a description of how it changes? It never hurts to contact the publisher and ask questions about the change and why it occurred. The publisher may have their name and contact details on the report or webpage. If there is no named contact then call the department's enquires number or send a message to their email address asking to meet or discuss the data.
      • +
    • How aggregated is the data? What is the number of real-world financial transactions that are expressed by a single line of the dataset you have? For budgets this will mostly be hard to tell - but with transactional expenditure you want to make sure that the data is fairly disaggregated. Ideally, each entry represents a transaction - but even if this isn't true you'll still want to ensure the number is not in the tens or hundreds of thousands (e.g. government programmes as a whole).
      • +
    • Ask for reference data. If your budget or spending data is augmented with reference data, make sure you have access to it. This might include functional or category codes on budget line items, location codes for describing recipient location, or codes that indicate the status of the record. 
      • +
    • Ask also for the guidelines people were given when creating the dataset. This will make it easier to understand what is included within the data, e.g. are the numbers in thousands / millions. 
      • +
    • Final tip: if the data you want is not given then narrow your scope. Your chances of success will be higher if you narrow the scope of the data you're requesting from the government and you are specific. Government is the de facto keeper of all kinds of data, so parameters that narrow your request are always helpful.
      • +

    An introduction to data-driven advocacy

    +

    Is going out and provoking a riot the best way to get a Government to take onboard your message? There are alternatives: hit them with the data hammer instead!

    +

    Making evidence-based policy proposals consists from three major phases: formulating your assumption, analysis (which often leads to re-formulating your assumption, and presenting your data in an engaging way in a policy proposal.

    +

    Analysing assumptions

    +

    Asking the right question is key to getting the most out of your data. We all make assumptions, and our organisation may have a particular standpoint on a given issue. Our first task is always to formulate our assumptions and then interrogate them ferociously. Although we try to be rational in this process, our judgement is often influenced by our subjective goals, values, and beliefs. Sometimes, you'll need to revisit your assumptions several times over to ensure they are valid and you can back them up with data. Once you know your policy problem is definitely a problem, you can work to package it in a way that's appropriate for your target audience. 

    +

    What is public interest? 

    +

    Often our job is to act in the public interest by analysing conflicting assumptions and working out which one is more valid. For example, in Greece, Spain, and many other European countries people protest almost everyday as the Government cuts spending to bring down its budget deficit. If the Goverment wanted to keep its current level of spending, but increased taxes to increase its revenue, different citizens groups would still protest depending on which taxes are to be increased. In any case, there will always be more than one interpretation of any Government policy, and interested side to support it, or not.

    +

    Policy analysis

    +

    Once we have a well defined policy problem, specific goals, or results different stakeholders are trying to achieve, and corresponding instruments they are using in this process, we may systematically search for the specific data needed to create our own policy proposals. This data can be obtained either from the Government, some other sources e.g. academic journals, private companies, or generated by ourselves. When data is gathered we will use a specific methodology to analyze it, and based on this analysis we will approve or reject our assumptions. If the assumption is rejected, based on our findings we will have to make the new assumption, and start the process from the beginning. If our assumption is approved, we will use our results to make a policy proposal to the Government.

    +

    Policy proposals

    +

    For CSOs it is important to recognize who is a decision maker, hence, who you should be targeting with your policy proposal. Policy proposals should be methodologically well structured, evidence-based, open for debate, and scientificaly evaluated. Governments will seldom take our policy proposals as their own policymaking, but may actually change its course of action, get new insights, views, and understanding of the subject. We may also use policy briefs to approach Government officials, or press releases to get the attention of the public. 

    +

    +

    Case study 

    +

    Fish subsidies

    +

    The influence CSOs have on government policy comes from a wide and varied set of activities. These can range from producing a widely shared dataset or infographic which subtly influences the mood of policy makers, to more targeted CSO advocacy and lobbying on issues they are experts.

    +

    The Fish Subsidies group (http://fishsubsidy.org) are a nice example of a CSO engaged in targeted activites. Having collected a comprehensive set of data on Fishing subsidies paid under the European Union’s common fisheries policy and they break this down into payments for every EU member state, and then complemented this with activites of fishing. They have produced a report (http://is.gd/XYPgq5) assessing the environmental and social impacts of the Financial Instrument for Fisheries Guidance between 2000 and 2006. This extensive document fed directly into the EU political decision making process.  

    5. Types of data

    +

    For the purposes of this manual, we have decided to abstract the meaning of the terms "budget data" and "expenditure data" to fit into a broader context. While these terms may have different meanings on a country by country basis, they are intended to be taken as defined in this section, throughout the rest of the manual. In this section, we look briefly at the two different types of data and what questions can be addressed using them.

    +

    +

    Budget data is defined as data relating to the broad funding priorities set forth by a government, often highly aggregated or grouped by goals at a particular agency or ministry. For instance, a government may pass a budget which contains elements such as "Allocate $20 million in funding for clean energy grants" or "Allocate $5 billion for space exploration on Mars". These data are often produced by a parliament or legislature, on an annual or semi-annual basis.

    +

    Spending data is defined as data relating to the specific expenditure of funds from the government. This may take the form of a contract, loan, refundable tax credit, pension fund payments, or payments from other retirement assistance programs and government medical insurance programs. In the context of our previous examples, spending data examples might be a $5,000 grant to Johnson's Wind Farm for providing renewable wind energy, or a contract for $750,000 to Boeing to build Mars rover component parts. Spending data is often transactional in nature, specifying a recipient, amount, and funding agency or ministry. Sometimes, when the payments are to individuals or there are privacy concerns, the data are aggregated by geographic location or fiscal year.

    +

    The fiscal data of some governments may blur the lines of these definitions, but the aim is to separate the political documents from the raw output of government activity. It will always be an ultimate goal to link these two datasets, and to allow the public to see if the funding priorities set by one part of the government are being carried out by another part, but this is often impractical in larger governments since definitions of programs and goals can be "fuzzy" and vary from year to year.

    +

    +

    Budget data

    +

    Using the definitions above, budget data is often comprised of two main portions: revenue and taxation data and planned expenditures. Revenue and spending are two sides of the same coin and thus deserve to be jointly considered when budget data is released by a government. Especially since revenue tends to be aggregated to protect the privacy of individual taxpayers, it makes more sense to view it alongside the budget data. It often appears aggregated by income bracket (for personal taxes) or by industrial classification (for corporate taxes) but does not appear at all in spending data. Therefore, budget data ends up being the only source for determining trends and changes in revenue data.

    +

    Somewhat non-intuitively, revenue data itself can include expenditures as well. When a particular entity or economic behaviour would normally be taxed but an exception is written into the law, this is often referred to as a tax expenditure. Tax expenditures are often reported separately from the budget, often in different documents or at a different time. This often stems from the fact that they are released by separate bodies, such as executive agencies or ministries that are responsible for taxation, instead of the legislature (http://internationalbudget.org/wp-content/uploads/Looking-Beyond-the-Budget-2-Tax-Expenditures.pdf).

    +

    Budgets as datasets

    +

    A growing number of governments make their budget expenditure data available as machine-readable spreadsheets. This is the preferred method for many users, as it is accessible and requires few software skills to get started. Other countries release longer reports that discuss budget priorities as a narrative. Some countries do something in between where they release reports that contain tables, but that are published in PDF and other formats from which the data is difficult to extract.

    +

    On the revenue side, the picture is considerably bleaker, as many governments are still entrenched in the mindset of releasing revenue estimates as large reports that are mostly narrative with little easily extractable data. Tax expenditure reports often suffer from these same problems.

    +

    Still, some areas that relate to government revenue are beginning to be much better documented and databases are beginning to be established. This includes budget support through development aid, for which data is published under the IATI (http://www.aidtransparency.net/) and OECD DAC CRS (http://stats.oecd.org/Index.aspx?DatasetCode=CRSNEW) schemes. Data about revenues from extractive industries is starting to be covered under the EITI (http://eiti.org/) with the US and various other regions introducing new rules for mandatory and granular disclosure of extractives revenue. Data regarding loans and debt is fairly scattered, with the World Bank providing a positive example (https://finances.worldbank.org/), while other major lenders (such as the IMF) only report highly aggregated figures. An overview of related data sources can be found at the Public Debt Management Network (http://www.publicdebtnet.org/public/Statistics/).

    +

    Connecting revenues and spending

    +

    It is highly desirable to be able to determine the flow of money from revenues to spending. For the most part, many taxes go into a general fund and many expenditures come out of that general fund, making this comparison moot. But in some cases, in many countries, there are taxes on certain behaviours that are used to fund specific items.

    +

    For example, a car registration fee might be used to fund the construction of roads and highways. This would be an example of a user fee, where the main users of the government service are funding it directly. Or you might have a tax on cigarettes and alcohol that funds healthcare grants. In this case, the tax is being used to offset the added healthcare expense of individuals taking part in at-risk activities. Allowing citizens to view what activities are taxed in order to pay for other expenditures makes it possible to see when a particular activity is being cross-subsidized or heavily funded by non-beneficiaries. It can also allow them to see when funds are being diverted or misused. This may not always be practical at the country level, as federal governments tend to make much larger use of the general fund than other local governments. Typically, local governments are more comprehensive with regards to releasing budget data by fund. Having granular, fund-level data is what makes this kind of comparison and oversight possible.

    +

    +

    What questions can be answered using budget data?

    +

    Budget expenditure data has an array of different applications, but it's prime role is to communicate to it's user broad trends and priorities in government spending. While it can help to have a prose accompaniment, the data itself promotes a more clear-cut interpretation of proposed government spending over political rhetoric. Additionally, it is much easier to communicate budget priorities by economic sector or category than it is at the spending data level. These data also help citizens and CSOs track government spending year over year, provided that the classification of the budget expenditure data stays relatively consistent.

    +

    Spending data

    +

    For most purposes, spending data can be interpreted as transactional or near-transactional data. Rather than communicating the broad spending priorities of the government like budget data should, spending data is intended to convey specific recipients, geographic locations of spending, more detailed categorization, or even spending by account number.

    +

    Spending data is often created at the executive level, as opposed to legislative, and should be more frequently reported than budget data. It can include many different types of expenditures, such as contracts, grants, loan payments, direct payments for income assistance and maintenance, pension payments, employee salaries and benefits, intergovernmental transfers, insurance payments, and more.

    +

    Some types of spending data - such as contracts and grants - can be connected to related procurement information (such as the tender documents and contracts) to add more context regarding the individual payments and to get a clearer picture of the goods and services covered under these transactions.

    +

    Opening the checkbook

    +

    In the past five years, there have been a spate of countries and local governments that have opened up spending data, often referred to as "checkbook level" data. These countries include, but are not limited to, the US (including various state governments), UK, Brazil, India (including some state governments) and many funds of the European Union.

    +

    Disclosure thresholds

    +

    At least two of these countries have imposed seemingly arbitrary thresholds on the size of transactions that are included. For example, the US and the UK exclude transactions under $25,000 and 25,000 GBP, respectively. Are these thresholds appropriate? That can't be known for sure without more information about how these numbers were arrived at. Principally, having thresholds or exceptions to the reporting of this data depends on the underlying systems that drive disclosure of this data. Are these systems linked directly with the accounting systems already used in the government, easing the burden of disclosure? If so, the threshold for excluding transactions should be very low (setting aside for a moment the cases that require redaction for privacy purposes).

    +

    If the systems are mostly divorced, as is the case with the US, then it begs the question, why? The more steps and processes between the internal government accounting systems and the public accounting systems, the higher the chance of error in the data and chance for omission of data. It also undermines the primary goal of public oversight if there are separate systems. However, governments often struggle with IT resources and contracting, which presents a tension between releasing any spending data at all, and release that is consistent with the above principles. If a threshold is necessary, then the amount should be consistent in size and scope with the overall expenditure level for that particular government. It is not appropriate, for example, that the threshold for the US State of Maryland spending reporting is also $25,000, when their annual budget is only a fraction of the federal government's budget.

    +

    Release early, release often

    +

    Spending data should be released in a relatively timely fashion, at least a monthly or quarterly basis. The timeliness of this data is what allows users to see if the spending priorities in the budget data are being reflected in the spending data. Also it allows the public and government stakeholders to view the current year's spending on a more detailed level as the next year's budget is being decided.

    +

    A good example of such release of spending information can be the Indian experience, especially the Employment Guarantee Programme, one of the major National flagship programmes on providing demand-based employment to the rural working age-group population in India. Its Management Information system (MIS) has become the most effective way of getting information on spending on a monthly basis. The data is updated monthly in an accessible spreadsheet format (Excel) at the sub-national government level. This makes the data transparent and available in the public domain to be equally accessed by all. The village level household database has internal checks for ensuring consistency and conformity to normative processes. It includes separate pages for approximately 250,000 local governments at the village level, 6,465 Blocks, 619 Districts and 34 States & Union Territories. The portal places complete transaction level data in the public domain.

    +

    However, problems related to maintaining an MIS in every state in a functional form and releasing continuous flow of data have been contentious issues. The major concerns emerge from lack in technical capacity as well as cost related issues. A cumbersome back-end system for supplying the data requires installation of a specific software with prerequisite configurations and technical operators with specific capacities. These requirements have raised costs and put a great demand on technology to ensure a continuous flow of data on the programme, specifically in most interior parts of the country and hence affect timely release of data.

    +

    What questions can be answered using spending data?

    +

    Spending data can be used in several different areas: oversight and accountability, strategic resource deployment by local governments and charities, and economic research. However, it is first and foremost a primary right of citizens to view detailed information about how their tax dollars are spent. Tracking who gets the money and how it's used is how citizens can detect preferential treatment to certain recipients that may be illegal, or if certain political districts might be getting more than their fair share.

    +

    It can also help local governments and charities respond to areas of social need without duplicating federal spending that is already occurring in a certain district or going to a particular organization. Lastly, businesses can see where the government is making infrastructure improvements and investments and use that criteria when selecting future sites of business locations. These are only a few examples of the potential uses of spending data. It's no coincidence that it has ended up in a variety of commercial and non-commercial software products -- it has a real, economic value as well as an intangible value as a societal good and anti-corruption measure.

    6. Getting and cleaning data

    +

    As you look into a new, data-driven project, you'll find it easy to imagine all the analysis and comparisons that could be done to prove a particular point or to explore some area of government activity. As you then look into the availability of data on a particular topic, disillusionment will quickly set in: information may be hard to find, unstructured or just not available to the public at all. As you search for data, there are a number of possible techniques you may want to investigate. Some are based on the governments cooperation while others use technology to bridge the gap between how information is shared and what is necessary for a data-driven advocacy project.

    +

    Getting data

    +
    Data processing pipeline
    +

    An increasing number of governments have accepted the need for pro-active transparency. When releasing open data, governments commit to making machine-readable information available on the web, in a form that enables anyone to use, re-use and re-distribute the data without legal or technical restrictions. Such releases should happen in a timely manner and include detailed, disaggregated data. Many countries have set up dedicated open data portals which provide easy access to all published datasets and relevant metadata (i.e. information about when a dataset was last updated, who published it and where documentation regarding format and contents can be found). The benefit of using open data when it's available includes the ease of access, but also usually a more authoritative base for any further analysis.

    +

    In some cases, open data is provided through an application programming interface (API), often a web-based method for retrieving, searching or even updating the available information dynamically. APIs provide up-to-date data in a granular and filtered form, removing the need to repeatedly process and update source files.

    +

    A common use case for APIs is relatively time-sensitive information, such as procurement calls and contracts which are released every day. In the UK, BusinessLink (http://www.contractsfinder.businesslink.gov.uk/data-feed.aspx) provides a number of data feeds which contain information about procurement notices. Similarly, the USASpending portal provides a set of APIs that can be used to retrieve up-to-date grants information for the US federal government (http://usaspending.gov/data).

    +
    If you are looking for a list of data catalogs from around the world, DataCatalogs.org is curated by experts in this area from around the world. Search for your country and see what you find.
    +
    +

    Freedom of information

    +

    Even before the rise of open data, many countries decided to increase the transparency of their governments by introducing freedom of information (FoI) legislation. Such laws enable every citizen to request documents and other material from parts of the government which do not merit special protection (e.g. due to concerns over privacy, national security or commercial confidentiality).

    +

    Journalists, activists and CSOs have long had channels of acquiring information. Sometimes, having a good relationship with a press-officer or a civil servant is good enough and making a formal request for information is unnecessary (your friendly press-officer may even feel slightly offended if you don't ask them nicely first). FoIs generate a lot of paperwork (hence grumpy civil servants), so if you do have the contacts, it may be a good idea to ask nicely first!

    +

    Freedom of Information requests often require some degree of preparation, so that the documents or databases that are requested are clearly identified, you know which department or unit is in charge of it and you can address possible concerns over privacy or commercial confidentiality in your request.

    +

    While freedom of information legislation is in force in many countries, it was often made before the need for structured data became apparent - thus many laws do not allow the citizen to specify a particular format. Many governments choose to release information on paper rather than in a structured, digital form, making the data processing step more painful. Still, the legally binding character of freedom of information requests often makes them an invaluable tool in the process of gaining access to financial data.

    +

    FoI requests may be necessary when you want to get more detail on the projects that government money is funding. Often the transactional spending data released will include only a brief description of the project, if at all. To get more information about it, you might need to submit an FoI request. For instance, if you have the high level payment information for a contract that includes the recipient, location and total amount, but you want to know the details of the contract deliverables, you will probably need to submit an FoI request for the full contract.

    +

    A good example of this process is the Sunlight Foundation's request for information on the Airport Improvement Program in the United States. The program accepts applications from airports around the country for infrastructure improvement grants, such as repaving a runway. Each project is assigned a safety priority rating and is prioritized in a queue. The high level spending information for this program was available in USASpending.gov, but since the priority ratings are specific to this program and not spending data in general, they were not included in that dataset. The Sunlight Foundation submitted a FoI request for the full dataset, including the priority ratings. After that, they were able to determine when airports with low priority projects were getting money, and how often. So the lesson is, if you see some interesting patterns in your high level spending data, don't be afraid to dig deeper and ask for more detailed program information.

    +

    Wanting to submit a request, but not sure where to start, who to address your request to or how to write it? Access Info (http://www.access-info.org/) are an organisation who work to help people obtain the information they require from the public bodies that hold it. They have also produced a toolkit (http://www.legalleaks.info/toolkit.html) to using FoIs. It's primarily aimed at Journalists, but most of the tips are equally relevant for CSOs.

    +

    Data scraping

    +

    Unlike open data or freedom of information requests, data scraping does not rely on the cooperation of government authorities for acquiring machine-readable documents. Scraping refers to the process of transforming unstructured documents - online database interfaces, PDF files or even printed documents into a more structured form (see section below on Optical Character Recognition for more tips on how to do this). Many tools and techniques enable such refinement by re-interpreting documents generated for humans into a more structured form which can be analysed, aggregated and distributed in new ways.

    +

    While there is an increasing number of easy-to-use scraping tools which do not require much technical knowledge, more complex processes - such as the automated scraping of thousands or millions of web sites or the mass interpretation of PDF files require some programming so that the process can be fully automated and produce reliable results.

    +

    In some cases, the only way to gain access to a set of figures is through the digitization of printed material. While scanners and optical character recognition (OCR) software can be used to import such documents, the high cost and low data quality generated through this approach often it an unattractive one.

    +

    A very accessible guide to scraping has been published (https://leanpub.com/scrapingforjournalists). Again, it brands itself at journalists (they're getting a little spoiled aren't they?), but scraping is scraping and so a worthwhile read for CSOs.

    +

    Another great source for tutorials, as well as being a tool itself to help with scraping is ScraperWiki. Using Scraperwiki (https://scraperwiki.com/) has the additional benefit of anything that you scrape being available to others, as most of the scraped data goes into a public data store.

    +

     

    +

    Getting data out of scanned documents

    +

    When you deal with scanned documents, the crucial step in the extraction process is to have the computer attempt to recognize any characters - letters, numbers and other signs. Optical character recognition (OCR) software is built to do this, accepting scanned pictures and PDF documents as an input.

    +

    There are both commercial software products for OCR (such as ABBYY FineReader, http://finereader.abbyy.com/), and some open-source software packages, such as Google's Tesseract (http://code.google.com/p/tesseract-ocr/). In general, the quality of all automatic recognition is limited, and you should make sure to cross check any numbers coming from scanned material against the printed documents.

    +

    Keeping the data around

    +

    As you retrieve data from the government (or other sources), it's easy to just consider the websites it has been released on as a permanent resource. Still, experience has shown that data does go away: whether it is through government re-designing its web sites, new policies that retract transparency rules or simple system failures.

    +

    At the same time, downloading complete copies of web sites - a process called mirroring - is a fairly well-established technique that can easily be deployed by civil society organisations. Mirroring involves an automated computer program (for a list see: http://en.wikipedia.org/wiki/Web_crawler) harvesting all the web pages from a specified web page, e.g. a ministry home page. In most cases, it is also possible to find old versions of web sites via the Internet Archive's Wayback machine (http://archive.org/web/web.php), a project that aims to create up-to-date copies of all public web sites and archive them forever.

    +

    Cleaning data

    +
    +

    Data Processing Pipeline

    +

    As you acquire government spending information, you will notice that such data often has many inconsistencies: program names are used inconsistently, financial amounts will be stated in badly formatted numbers, while some data may not be usable at all due to file corruptions. In short: data always needs to be cleaned and processed. In fact, processing, augmenting and cleaning the data is very likely to be the most time- and labour-intensive aspect of your project.

    +

    Processing stages for data projects

    +

    While there are many different types of data, almost all processing can be expressed as a set of incremental stages. The most common stages include data acquisition, extraction, cleaning, transformation, integration, analysis and presentation. Of course, with many smaller projects, not each of these stages may be necessary.

    +

    In this process, each stage fulfils a unique purpose

    +
    • Acquisition describes gaining access to data, either through any of the methods mentioned above or by generating fresh data, e.g through a survey or observations.
      • +
    • In the extraction stage, data is converted from whatever input format has been acquired (e.g. XLS files, PDFs or even plain text documents) into a form that can be used for further processing and analysis. This often involves loading data into a database system, such as MySQL or PostgreSQL.
      • +
    • Cleaning and transforming the data often involves removing invalid records and translating all the columns to use a sane set of values. You may also combine two different datasets into a single table, remove duplicate entries or apply any number of other normalizations.
      • +
    • Analysis of data to answer particular questions we will not describe in detail in the following chapters of this book. We presume that you are already the experts in working with your data and using e.g. economic models to answer your questions. The aspects of analysis which we do hope to cover here are automated and large-scale analysis, showing tips and tricks for getting and using data, and having a machine do a lot of the work, for example: network analysis or natural language processing.
      • +
    • Presentation data only has impact when it is packaged in an appropriate way for the audiences it needs to aim at.
      • +

    As you model a data pipeline, it is important to take care that each step is well documented, granular and - if at all possible - automated. This is particularly true when processing more complex datasets, such as transactional government expenditure which has been scraped from a government website.

    +

    Data provenance

    +

    Good documentation on data provenance (the origin and history of a dataset) can be compared to the chain of custody which is maintained for criminal investigations: each previous owner of a dataset must be identified, and they are held accountable for the processing and cleaning operations they have performed on the data. For Excel spreadsheets this would include writing down the steps taken in transforming the data, while advanced data tools (such as Open Refine, formerly Google Refine), often provide methods of exporting machine-readable data containing processing history. Any programs that have been written to process the data should be available when users access your end result and shared as open-source code on a public code sharing site such as GitHub.

    +

    Tools for documenting your data work

    +

    Documenting the transformations you perform on your data can be as simple as a detailed prose explanation and a series of spreadsheets that represent key, intermediate steps. But there are also a few products out there that are specifically geared towards helping you do this. Socrata is one platform that helps you perform transforms on spreadsheet-like data and share them with others easily. You can also use the Data Hub (pictured below), an open source platform that allows for several versions of a spreadsheet to be collected together into one dataset, and also auto-generates an API to boot.

    +
    +

    Normalizing data

    +

    Data that comes from the government is often generated across multiple departments by hand. This can result in inconsistencies in what kinds of values or formats are used to describe the same meaning. Normalizing values to be consistent across a dataset is therefore a common activity.

    +

    First, you want to start by finding all of the distinct ranges of values for the different columns in your dataset. You can accomplish this by using a database query language (such as SQL's DISTINCT), or by simply using the 'filter' property on a spreadsheet program.

    +

    For example, if you have a spreadsheet with contracting data, and one column is 'Competed?', you would expect the values to be 'yes' or 'no'. But if this spreadsheet is an amalgam of spreadsheet data from multiple users and departments, your values could vary among the following: 'Y', 'YES', 'yes', 1, 'True', 'T', 't', 'N', 'NO', 'no', 0, 'False', 'F', 'f', etc. Limiting all of these potential values to two clear options will make it easier to analyse the data, and also easier for those who follow in your footsteps.

    +

    Especially with financial data, numbers can be formatted several different ways. For example, are your negative values represented with a '-' or placed inside '( )' or possibly even highlighted in red? Not all of these values will be easily read by a computer program (especially the color), so you'll want to pick something clear and consistent to convert all your negative values to (probably the negative sign).

    +

    Is all your numerical data measured out in ones or is abbreviated in thousands? Especially with budget data, order of magnitude errors are not uncommon when one department thinks they're reporting in thousands or millions by default but others expand their data all the way to the ones place. Are some values in scientific notation (e.g. 10e3938)? Make sure all your values are consistent, otherwise your analysis could contain serious errors.

    +
    A column of data requiring name normalization
    +

    Data quality

    +

    Data cleaning and data quality go hand in hand. You may find it helpful to conduct data quality tests after your analysis in order to measure your confidence in the data, or before to decide whether you want to do the analysis at all. Data quality can mean a few different things but here or some key factors to consider.

    +

    Completeness

    +

    Is all your data there? If you have a time series of some data, do you have approximately the same number of rows for each year or are there major outliers? An extremely high or low number of rows that looks out of place may indicate duplicate or lost rows. Or it could indicate a change in how data was reported at that time. Are there values for each column where values are required (according to the documentation)?

    +

    Accuracy

    +

    Does your dataset match up with other available data out there? This can be difficult to confirm, but it is often useful to employ a sampling methodology. Select a significant sample of your records (3-5%) and decide which values you can check for accuracy. If your data is transactional contract data, you could request the full contracts for your sample records and compare the structured data you have with the prose of the contract to confirm that it is correct. You can sometimes check contract and grant amounts from journalistic outlets and even company websites.

    +

    Timeliness

    +

    Sometimes governments can lag behind in their reporting. Even if a dataset says it's for a specific time period, do you notice a long tail of records closer to the end of the time frame? If so, you may want to restrict your analysis to the subset of data where you have a healthy sampling.

    +

    Case studies

    +

    Some projects have gone beyond simple assessments of data quality to ensure confidence in the underlying result of a project, but are projects completely about data quality themselves!

    +

    The Open Knowledge Foundation recently created a data quality monitor for UK transactional expenditure data (http://openspending.org/resources/gb-spending) which has been developed in cooperation with the data.gov.uk team. Each imported file is documented, highlighting any mistakes in the data structure or formal representation of the data. More information at:

    +

    The Sunlight Foundation has run a yearly report called Clearspending (http://sunlightfoundation.com/clearspending/) for the past three years that assesses the quality of the data reported in the US federal spending dashboard, USASpending.gov. The results haven't been very good but have highlighted the need for data standardization and oversight of the data release process.

    +

    As an even more thorough technique, little programs and tests can be written to check data validity along the processing pipeline by testing certain assumptions about each record in the dataset. If you do not have the option of creating such tests, you may still be able to create a spreadsheet in which you record known defects and issues with the data, including source files that did not open and inconsistent column values.

    +

    The most powerful data quality tool is much more simple, though: sharing the data. Without access to your source data, others will have to trust your word blindly. Therefore, sharing both the source and processed forms of your dataset is essential to discussing data quality.

    +

    Categorization and reference data

    +

    One of the most powerful ways of making data more meaningful for analysis is to combine it with reference data and code sheets. Unlike transaction data - such as statistical time series or budget figures - reference data does not describe observations about reality - it merely contains additional details on category schemes, government programmes, persons, companies or geographies mentioned in the data.

    +

    For example, in the German federal budget, each line item is identified through an eleven-digit code. This code includes three-digit identifiers for the functional and economic purpose of the allocation. By extending the budget data with the titles and descriptions of each economic and functional taxonomy entry, two additional dimensions become available that enable queries such as the overall pension commitments of the government, or the sum of all programmes with defence functions.

    +

    The main groups of reference data that are used with government finance include code sheets, geographic identifiers and identifiers for companies and other organizations:

    +

    Classification reference data

    +

    Reference data are dictionaries for the categorizations included in a financial datasets. They may include descriptions of government programmes, economic, functional or institutional classification schemes, charts of account and many other types of schemes used to classify and allocate expenditure.

    +

    Some such schemes are also standardized beyond individual countries, such as the UN's classification of functions of government (COFOG) and the OECD DAC Sector codes (http://www.oecd.org/dac/aidstatistics/dacandcrscodelists.htm). Still, the large majority of governments use their own code sheets to allocate and classify expenditure. In such cases, it is often advisable to request access to the code list versions used internally by government, including revisions over time that may have changed how certain programmes were classified.

    +

    A library of reference data that can be re-used across different projects and it is a valuable asset for any organization working with government finance. Sharing such data with others is crucial, as it will help to enable comparable outputs and open up options for future data integration. Existing repositories include the IATI Standard (http://iatistandard.org/) and datahub.io.

    +

    Geographic identifiers

    +

    Geographic identifiers are used to describe administrative boundaries or specific locations identified in a dataset. While some regional classifications (such as the EU NUTS) are released on the web, there is also an increasing number of open databases which contain geographic names - including geonames.org and the recently developed world.db.

    +

    Another related technique is the process of reverse geo-coding: translating a human-readable address into a pair of coordinates. Services like nominatim (http://nominatim.openstreetmap.org/) will not only enable users to generate precise maps of projects in a region, they will also return the responsible administrative boundary for many coordinates. This means that projects which are given by precise address can also be aggregated by state, region or any other geographic unit.

    +

    Additionally, many countries have shapefiles of their political and geographic districts available (usually through the census or interior bureaus) that can be imported into custom mapping applications, like TileMill (http://mapbox.com/tilemill/).

    +

    Company and organisational identifiers

    +

    As you look into spending data that includes recipients outside the government, you'll find companies which act as suppliers to government, but also other types of entities including charities, associations, foreign governments, political parties and even individuals which act as recipients of direct assistance.

    +

    Identifying such entities is notoriously hard, since the only information kept by government is often a simple name (which may not uniquely identify the beneficiary, for example "MS"). While most (if not all) countries maintain company registers which assign some type of unique identifier to a company, these databases are often not accessible in bulk and not used coherently across different parts of government. Alternative identifiers - such as tax identifiers and company IDs from private business information suppliers (such as Dun & Bradstreet in the US) - further complicate this process.

    +

    As an alternative, open registries are beginning to compile organisational identifiers in a form that is easy to re-use and thus enables the sharing of databases which have been augmented with such information. OpenCorporates.com (http://opencorporates.com) is a startup that collects information from companies world-wide and provides a convenient API to match datasets with the list of known countries. The IATI project for aid transparency is working towards similar standards for other organisations, such as foreign governments and charities active in the development space.

    7. Analysis

    +

    +

    Data Processing Pipeline

    +

     

    +
    Dear reader - we know you are the experts. We're not going to teach you all the basics, but this section's aim is to show you how technology can help speed up, scale up or simplify your working practices. For this reason in this section we will say only a few words about statistical methods, even though many of you will be very familiar with those.
    +

    Once you have your data in a suitable format to be dealt with, much of your analysis can be automated with a custom program or inside spreadsheet software. Obviously the deeper your data dive is, the less automated things become, but having a standard set of post-processing, analytical steps to run on a dataset helps make sure you haven't missed anything. Some of the steps may seem basic, but it always helps to have a checklist!

    +

    Sorting data

    +

    Possibly the simplest way to find some interesting points is to sort by a column or category. In spending datasets, it's common to rank data by the largest transactions and look at the beneficiaries. This also gets you the maximum or minimum values of a column, which could be interesting.

    +

    Basic statistical analysis

    +

    Statistical analysis such as calculating percentiles or the average of a dataset, are essential to characterize your data. These are available in most programming languages as well as most spreadsheet programs like excel. A more complete list of statistical tools and resources can be found at the resources chapter.

    +
    • An average is a measure of central tendency in a sample. It can be described as the "typical" or representative value of a data set. Knowing the average helps detect whether any single payment is atypical.
      • +
    • A percentile is the value of a variable under which a certain percentage of the data fall. For example, if a transaction amount is in the 90th percentile, then it is larger than 89% of the transactions in the dataset.
      • +
    • A standard deviation measures how far away a value is from the average. If your standard deviation is low, then it means your row values are clustered together. If it's high, then they are spread out. By checking for values that are one, two, or three standard deviations outside the mean, you can detect outliers. Outliers could be errors or extremely interesting data points.
      • +

    Summarizing or aggregating by categories

    +

    Sometimes data is available at a very high level of detail, making it hard to see the big picture. It's a good idea to have a standard set of categories to aggregate by, such as geographic unit, functional classification, account number, fiscal year or other temporal unit, government department, or even recipient.

    +

    Normalization for comparability

    +

    In order to be able to compare data across time or other parameters, you may need to systematically adjust the data values. Here are some examples.

    +
    • Deflation or inflation. Expenditure data is naturally obtained with current (or nominal) money value, but such data is not directly comparable across different years since inflation affects the value of money. In order to compare currency across years, it is necessary to deflate or inflate the value according to a base year. For example if we wish to compare the budget of a program in dollars from 2000 with the same program in 2012 we must adjust the value of the 2000 data and the 2012 dollars to a base year. Deflator and inflator factors are calculated by central banks in most countries and a comprehensive list by country can be found at the World Bank website (http://worldbank.org)
      • +
    • Scale and normalize values. When we wish to compare patterns of budget programs sometimes the difference in scale between budgets may prove difficult to display in a graphic form. In such cases is useful to compare percentages instead of absolute numbers. For instance, it is very common to express budget and spending data as percentages of a country's GDP, instead of in absolute numbers. If you're mapping spending, make sure to scale your data by population density. Otherwise, your map will most likely just be a map of where your populace lives.
      • +
    You may want to visualize your data to get better understanding of your initial assumptions. This may save you a lot of time in case your initial assumptions were not good, and may also help you to get more insight into new findings your data might contain but previously were not clearly visible. For this purpose you can use scatter plots, line graphs, heat maps, bar charts.

    8. Using technology in your work

    +

    As you move into more advanced data projects, you may find yourself in a situation where a research project turns into an effort that involves coders, designing databases and web sites. It is important that you take a step back and realize that you are now running not just an advocacy effort, but also an IT project.

    +

    There are many difficulties which CSOs face when developing software. Some common issues include:

    +
    • Difficulties in finding qualified developers that want to contribute to your projects at a reasonable rate, as well as in the communication between CSO staff and developers.
      • +
    • Clearly communicating the requirements for the software so that both non-technical and technical staff share a vision for the outcome of the project.
      • +
    • The estimation of time and resources for particular tasks, especially how to handle projects that drastically overrun the timeframe and funding they were initially assigned.
      • +
    • Evaluating the work of developers to ensure that the product that has been delivered is according to what has been agreed, especially in small projects with only a single coder or when working with external contractors.
      • +
    • Maintaining the project after the main development period has finished
      • +

    Starting from scratch vs. re-using Components

    +

    It is always going to be more costly and riskier to develop something from scratch than to customise something that already exists.

    +

    You should make the software behind your projects open source. If many other organizations do the same, this allows code to be reused across jurisdictions. Not only does this ease the financial burden, but it helps create the expectation in populaces around the globe for the high quality engagement tools that their neighbouring country has access to.

    +

    That's not to say you should never develop something new - just ask around first, and make sure that what you are asking is technically feasible.

    +

    Commissioning New Software

    +

    Purchasing software is more closely related to having a piece of clothing made than to buying chairs. You can give the designer a basic vision of what you would like, but you will always need to come back to make sure it really fits, and your thoughts may change when you see things in practice. If you have an arrangement with your tailor which allows you to first specify the general idea, and a couple of other appointments for fittings and trials, you'll probably end up with a better and more creative result than if you tried to design the whole thing and it was simply unveiled to you at the end. You'll also feel more in control and it may even be quicker to do design and implementation in parallel.

    +

    Defining requirements & Designing the solution

    +

    Define the basic components of your project and prioritise them by their importance. As the developers start working on one of these chunks, you can then break it up into more specific tasks based on your evolved understanding of the project. A popular technique for this purpose are user stories, small narrative pieces that describe each problem: "As a [web site visitor] I want to [be able to see a supplier's contracts] so that I can [understand what services they provided to government]". The key to these stories is that they describe the actual user need, not the details of the solutions that you have envisaged. While you should of course discuss those with the developers as well, defining solutions is mainly the job of the developers, not the project manager.

    +

    Implementation: iterations and milestones

    +

    A saying amongst developers goes: "Walking on water and developing software from a specification are easy if both are frozen." As your software project is progressing you will likely realize that the specifications you have given need to be revised or extended. Yet by modifying the requirements you are essentially shifting the ground on which developers are executing - meaning they will have to stop their work to adjust. To prevent such changes from freezing all development, the process of introducing changes and additional requirements needs to be structured.

    +

    Iterations are periods of a defined length - often two or four weeks - during which developers are tasked to execute a set of previously selected user stories or requirements. Before the iteration starts, developers have to pull in the work from a list of tasks (a so-called backlog) prepared by the project manager, committing themselves to delivering those tasks within the agreed period. Crucially, project managers are not allowed to extend or revise the scope of an iteration while it is ongoing (unless they want to declare it failed). This method ensures that changes are introduced in bulk and understood by the team. This approach mandates the opposite of the more common unstructured communication between managers and developers, e.g. emails with unsorted lists of change requests which tend to be ignored and lead to confusion.

    +

    Whenever you consider an additional requirement, be sure to consider if it is realistic within the resources you have available. "Scope creep", the progressive extension of a project during its development, is a common cause of project failure. By becoming more and more ambitious, the project finally ends up with no usable product at all. To avoid scope creep, make sure to have a storage area for long-term ideas. Also make sure that developers accept additional tasks through a pull process, and not by having them pushed into their workflow.

    +

    Maintenance

    +

    Make sure to budget a for ongoing maintenance after the end of your project. Who is going to guarantee that the servers stay online? Who is going to fix a typo? It is unlikely that your project will remain entirely static after its initial development, so you should have an explicit agreement with the developers regarding future support. It is also useful to collect feedback after the projects release to commission a small number of additional days when enough additional work has accumulated.

    +

    Roles and how to find developers

    +

    The key ingredient to a successful software development project is having the right people on staff or as contractors. Depending on the scope and type of your project, you may need a variety of skills - these are some of the common descriptions:

    +
    • Web designers typically produce designs and layouts for web pages, often initially in a graphics program like Adobe Photoshop. Most, but not all, web designers then translate their designs into web markup (HTML, CSS).
      • +
    • Web developers are more technical. They produce interactive web interfaces such as search masks, browsers or specific form-based operations. They often use programming languages such as PHP, Ruby on Rails, Python or JavaScript.
      • +
    • Visualization designers develop graphics that represent quantitative information. A key distinction here is between non-interactive graphics (i.e. static images) and interactive visualizations, which often require some programming. There are still very few designers who design interactive visualizations, so rates may be relatively high.
      • +
    • Software developers are even more technical, developing backend software for data processing or acquisition. They are experienced in the use of database software (such as SQL databases) and programming languages such as Ruby, Python or Java.
      • +
    • Data scientists and statisticians produce analysis based on large sets of data, detecting tendencies and outliers in the dataset. They are not usually expected to produce front-end applications, but may produce software in the process of analysing data.
      • +
    • Usability experts and user experience (UX) designers think about the way your user will interact with your site answering questions such as 'is it obvious from the landing page what the purpose of this site is?'
      • +
    • Testers try and break things to test their robustness. This is particularly useful e.g. if you think your project will receive a lot of traffic as a result of a media campaign, you want to know your site can survive the hit
      • +

    Good places to look for developers

    +

    The easiest way to meet developers is through community meetups, such as hackdays. During such events, coders meet up to cooperatively develop prototypes of new software. To meet volunteer developers who can help you make sense and unleash the power of government spending and budgets, it's wortwhile to investigate events such as Random Hacks of Kindness (http://www.rhok.org), Data Kind (http://datakind.org/) and TechCamps (http://techcampglobal.org/).

    +

    There are a few ways you can discover if there is a hackday in your area. One way is to search on Lanyrd (http://lanyrd.com/search/?q=hackday&context=future) or set up an account on that system and request that you are alerted when there is a hackday in your area. Another approach is join mailing lists of organisations that might help you find developers e.g. the Open Knowledge Foundation lists (http://lists.okfn.org/mailman/listinfo) or the Sunlight Labs mailing list (https://groups.google.com/forum/#!forum/sunlightlabs).

    +

    What do things cost?

    +

    It's impossible to give concrete guidelines on how much a project should cost. Developers' salaries are generally quite high for a country's average, but vary very strongly from country to country. Worried about your project spiraling out of control? We'd recommend agreeing on a price per iteration, and it may be a good idea to draw up a contract which allows you to break it off if you are not happy with the work at the end of an iteration. Plus, you can generally also find a friendly developer to glance over a quote from a company for a sanity check.

    Presentation and engagement
    9. Presentation and engagement
    10. Selecting methods and tools

    9. Presentation and engagement

    +

    While raw financial data is valuable just as a tool for research, analysis and investigation, it can also be directly presented to an audience in a wide variety of ways. Such presentations can take the form of press releases in which data is used to make an argument, interactive infographics which condense information to visually represent a key finding or even searchable on-line databases that allow each user to find spending in their neighbourhood.

    +

    +

    Clarifying your topic

    +

    "It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away" - Antoine de Saint Exupéry

    +

    If you are reading this, you probably already have a fairly clear idea of the topic that you want to investigate and present. Still, it is useful to look at different ways of defining your subject. In some cases, you may merely want to give an overview of a particular area, e.g. by showing the proportions of government spending which go into the various functions of government. In other cases you may have a particular hypothesis that you want to portray or a particular journalistic story that you mean to tell, such as a case of systemic corruption that becomes evident only as you contrast one department's spending patterns with that of another one.

    +

    Sometimes there is a temptation to get lost in complexity. Using all the data and tools available to you to their full potential can force an almost endless amount of information onto our audiences. The benefit of doing so is limited, however: people will not understand the point that you are making, or, even worse, ignore you.

    +

    At the same time, technology can also be used to do the opposite: focus in on the key points, hiding other detail. Selecting where to go into depth doesn't have to be the author's responsibility any longer, it can be a choice that the user of data-driven applications is empowered to make based on their own interests. The prerequisite for using technology to create a clear focus is answering the question: What is the core of what we mean to present, what is the essence of the narrative our project is intended to convey?

    +

    Setting an objective

    +

    As you focus the topical core of your project, you also need to think about the objectives of your project: is it intended to inform the general public about an aspect of government budgeting or spending practices? Are you making a specific case to government, e.g. by proposing a new policy which you have gathered evidence and developed arguments for?

    +

    Your project can also serve as a forum for discussions, which would usually require you to facilitate the debate to maintain ongoing engagement.

    +

    The objective may be to establish a network-based, collaborative effort, e.g. to build a community that may review documents released by the government or evaluate the conditions of public contracts. Such modes of engagement require extensive design to encourage users to contribute in a way which best matches their particular skills and interests.

    +

    Some example of working with the public to collaboratively clean up financial data is Linked Gov (http://linkedgov.org/). This is a platform that will allow the public and civil servants to access and do small tasks that will clean government financial data.

    +

    Targeting an audience

    +

    The efficiency of projects based on spending data depends heavily on how well you have defined and understood your audience. If your message is too general for a specialized audience, it will be ignored. At the same time, a complicated and very technical presentation will usually fail to address broader parts of the population. Whether it be that your main audience is a specific constituency, government officials, the general public or a combination of those, it is important that the media and methods you apply help to get the message(s) across to the audience that you are targeting.

    +

    The general public

    +

    Is the purpose of your campaign to stimulate discussion amongst the general public? Unless your organization has exceedingly well-developed outreach channels, you will not usually be able to address the general public directly. Still, thanks to the social media and internet-based direct communication, many organizations can now contact and involve citizens who have shown some interest in their work directly. In cases where your main audience is the general public, simplicity is a must. It is necessary that the vast majority of people get a grasp of what you are trying to communicate. In these cases you may need to emphasize the visual presentation of the data that gets quickly and succinctly to the point, such as infographics. See the case study at the end of this chapter for how an organisation in Nigeria used Social Media to reach out to the general public.

    +

    Mass media

    +

    Unlike direct channels, broadcast and print media require you to first convince an intermediary of the relevance and quality of your message. Media are often the best channels to use when we want to create social pressure on governments to change some budget issue.

    +

    Community and constituency

    +

    This may include people and organizations with which you have previously cooperated or which are active in the same domain. Unlike the wider public, they are more likely to be interested not just in your conclusions, but also in the data and tools that you have used to reach your conclusions. Creating an open space for your community to discuss your work is essential in getting relevant feedback. When you need to reach to your constituency it may be necessary to provide a broader context, such as a full report including methodology, conclusions and possible paths of further discussion.

    +

    Government

    +

    Officials are often a target of your outreach, since they are in the position of actually deciding and implementing changes in policy. While they can be reached through public pressure and the media, your organization or some of its constituency may also choose to contact governments directly. In the case of trying to reach government officials, a balance of the previous approaches is very important. Meetings with government officials are rare and short-lived. Condense your main argument and supporting evidence into one page or less, and include a clear action that you want them to take.

    +

    Obviously, on any given issue you may choose to involve any set of these groups, but it is still important to have a clear picture of who they are and what their incentives motivate them.

    +

    +

    Case study

    +

    BudgIT Nigeria on social media

    +

    Data is invaluable. To the uninformed person where information asymmetry thrives, access to data illuminates the path to facts and provokes emotions that trigger results. Nevertheless, poor handling of data puts valuable facts in an opaque structure that communicates nothing. Data could be in a maze of thick document riddled with complex terms or iterations of figures that doesn’t connect the user. Not being able to promote discussion nor give provide contextual understanding, data may could be worthless.

    +
    +

    Nigeria returned to democracy in 1999 after lengthy years of military rule. Data under the barrel of the gun was an exclusive preserve of rulers as probing the facts behind data is taken as an affront to authority and strive to question the stained reputation of the junta. Civil servants were bound by the Official Secrets Act not to share government information hereby putting the citizens in the dark. Even after thirteen years of return to democracy, there is clearly a gap in accessing public data with some government officials still stricken with military era hangover. Data especially in terms of public expenditure communicates little to the larger sections of public who are not versed in financial accounting nor have requisite understanding to evaluate the complex arithmetic.

    +

    BudgIT, creative start-up sees a huge opportunity in using creative data visualization to stimulate interests concerning public expenditure. Understanding the ubiquity of the mobile device within the Nigerian locality and the increasing number of Nigerians online, BudgIT sees the opportunity to engage Nigerians and explain public expenditure in a simpler way. This thrives on building engagement across all platforms, encouraging the community to action via NGOs and reaching out to everyone to promote citizen participation. This is about making public data a social object and building an extensive network that demands change.

    +

    Building The Community

    +

    Across our interactions with users, we see a gulf in understanding what the budget is and what the citizens usually expect. We have engaged over 10,000 Nigerians over the budget and we profile them into three to ensure optimum value is delivered. The categories are briefly explained below:

    +

    Singular Users: These are users who want information in simple and quick format. There are interested in data but not in a deeper scale of analyzing the content or probing further. A series of simple tweets is enough for them to retell the stories or interactive applications that give a snapshot.

    +

    Closed Loop Users: A set of users who stimulate a discussion pattern, engage the data channel effectively to increase their knowledge of subject matter or challenge the assumptions of data. These set of users are adherents of the platform via social media, hence they ensure in building the platform through feedback systems or referral to their social connections.

    +

    The Data Hog: This set of users want the raw datasets to rework visualization or do extensive analysis for personal or enterprise purposes. We simply give them the data for their definitive purposes.

    +

    Engaging the Citizen

    +

    Across every society lies a literacy span and engaging every component is highly critical to the societal growth and stability. There is always a storyboard in every life that volumes of data can be matched with. People constantly want to be more informed especially concerning issues that they find difficult to understand. Engaging citizens is to take critical analysis of the target users at a time and itemize the possible profiles. A look at user profile demands a thorough analysis of their empathy, attention and insight towards the data available to them. What does the Nigerian citizen care about? Where is the information gap? How quickly can we reach out to them and place data in the storyboard of their lives? A critical understanding of the user’s psychology and the perceived response to the data is the first needed level of analysis. BudgIT’s immediate reach is to the average literate Nigerian connected to online forums and social media. Most online users amidst the array of interests in gaming, reading and sharing social connections within a limited timeframe will definitely need data in a brief and concise manner. After a snapshot of data either as a tweet or infographics, there’s an opportunity to build linked data on other platforms where the big picture can be set and interaction can be enhanced. <p >An important angle of visualization to us is understanding the data appreciation level of the users. Complex diagrams, superb infographics and aesthetic interactive applications might not convey the exact meaning to user based on his/her previous approach to data. Data vizualization needs to take into consideration how users can easily grasp the vizualized data and subject it to personal interpretation. A good visualization transfers knowledge and mostly important brings forth a story the user can easily connect with.

    +

    For us in BudgIT, our engagement model is anchored on the following:

    +
    • Stimulating discussion around Trends: In engaging with users on public data, BudgIT keeps track of online and offline discussions and seeks to provide data to enliven the interactions. A glaring example was the fuel strikes in January 2012 where there was a constant agitation among the protesters on the need to reduce the size of governance. BudgIT tracking the discussion via social media in 36 active hours quickly built an app that allows citizens to cut the Nigerian budget. The huge response with over 3,000 users who interacted with the budget using the app refined our engagement model. We keep looking for trends in the polity and matching it with relevant data quickly rendered into tweets or infographic display that quickly  extends our influence.
      • +
    • Constructive feedback mechanism and Balanced outlook: Data speaks volumes and individuals subject it to personal interpretations. In the engagement with users, feedback is enabled through discussion boards or retweets. Most users throw up discussions that tend to ask about stories behind the data and seeking opinions of BudgIT. Its of utmost priority to ensure that opinions only explain the facts behind the data and does not conform to individual disposition to the subject matter. It’s most necessary to build up feedback channels and engage the users creatively to ensure the community built around the data is sustained.
      • +
    • Localize Outlook of Data: For a dataset targeted at a particular group, BudgIT is building competency to localize its content and promote a channel of discussion that connects to the users. This involves taking a cultural outlook at the icons, symbols, objects and language to ensure that engagement concerning the budget flows seamlessly. This phase of engagement is with the grassroots who mostly don’t have access to Internet but only possess SMS-based mobile phones.
      • +

    After making the public expenditure data available in an easy-to-read format as shown on our portal (http://yourbudgit.com), we reach out to the citizens through the civil society organizations to ensure that citizens monitor capital projects in the budget. We also plan to develop a participatory framework where citizens and government institutions can meet in town halls to define key items in the budget that needs to be prioritized. Once we get citizens to be aware of capital projects in the budget and connect them with civil societies where BudgIT is not located, citizens can track report projects and report status. Ensuring that citizens of any literacy span are armed with data and possess a clear path to demand action, BudgIT is crossing the rubicon from open data to open action.

    10. Selecting methods and tools

    +

    The key to using data to tell people about your issue is to choose a really good story. The key to finding a good story is to formulate the right question for your audience.

    +

    Be guided by what interests you, as this stands a good chance of interesting other people. If you don't feel confident about your judgement then try lots of things that you consider interesting and see if it makes impact on the people you care about. Are they sharing your work on social media? Do they email you or link to your work from their own sites? If the answer is yes then you are really making good data stories.

    +

    Of course rigour and accountability are essential and you should always include links to your methodology and data, but it doesn't need to obscure your headline. Telling a story is the difference between people not reading your work and making it all worth the effort.

    +

    If you've already got data products that you want to share, try disseminating them in a couple of different ways. In addition to the methods outlined below, try preparing primers or manuals for other groups that want to start working with datasets that you may be particularly experienced with. You can also organize workshops, meetups, and webinars to familiarize groups with the data you work with. If grassroots issue groups can internalize your data products and present them in a contextually relevant way to their user base, then both groups have gained something from the partnership.

    +

    This is a collection of (hopefully) inspiring examples where groups have connected to the public directly through their websites, social media, or through the mainstream media. If you are interested in making media but don't have the capacity then fear not! Check out the first section of the book on reaching out to other organizations for help or running your own project.

    +

    Communicating via the web

    +

    If you are creating or adding to a website that you want citizens to access directly then here are some of the ways you can really use budgets or spending data to increase the value of your organisation.

    +

    Firstly we highly recommend that if you have some really great data, that's a story in itself, then don't waste any time - put the data up on your website and write a great story to go with it. Make sure the full dataset is available to download, ideally in a couple of formats like csv and json, and explain your methods. Then encourage your supporters to use and play with the data, add comments and talk in your community site or social networks.

    +

    If you feel your data needs that little bit more work to make it interesting or digestible, here are some examples of portals, interactive infographics, images, reports, and a searchable dataset.

    +

    Infographics and images

    +

    The art of the infographic is to make the complex look simple. This can be invaluable if you can communicate huge amounts of data in one picture. It is always recommended, both as a boost to your credibility and to allow others to extend your work, to publish the complete data needed to recreate the image. This allows your community of supporters to create their own visualisation and check your work easily before republishing it.

    +

    Below is an infographic summarizing the history of the US Federal Budget, by the Congressional Budget Office (CBO) /http://www.cbo.gov/publication/42636). It shows the budget over time and as a percentage of GDP, and in the context of total spending and revenue. This was used to emphasise that the federal government's budget deficit for 2011 was the third-largest in the past 40 years:

    +

    +

    Images of spending

    +

    This "binder full of burgers" infographic was created to show the sandwich of spending for Barack Obama's 2012 presidential election campaign. The data is not especially novel but the graphic is fun and engaging.

    +

    burger full binders

    +

    Interactive infographics

    +

    An interactive infographic can add even more data to a single screen full of images. A simple example is a bar chart where hovering over a bar with the cursor results in a dialogue box that tells more detail, like the exact value of that bar.

    +

    It seems there are two types of interactive graphics that work very well. One is where the user finds their own path through a complicated landscape of data. For example a map of the world where you can zoom in to details of each country or regions within a country and discover some facts about that area. The other is where the infographic guides the user through a story. For spending data in particular, people often like the ability to "drill down" in an infographic, or in other words, start with a high level view of the data and then gradually view more detail about different sections that they can select.

    +

    In recent years we have seen the quality of interactive infographics grow and become easier to produce. Anyone with a Google account can now make maps (http://www.google.com/drive/start/apps.html#fusiontables) and charts (https://developers.google.com/chart/). Equally, the tools to manipulate and display data in more sophisticated ways have become more accessible with efficient and usable javascript libraries. Using these libraries requires a bit of technical skill, so you may want to reach out to other organizations, as described in the first section of this book.

    +

    Budget interactives

    +

    An interactive graphic gives you the perfect opportunity to allow people to choose their own budget. This interactive chart called Citizen Budget is from the non-profit Open North. It shows Canadian budgets and gives citizens the chance to play at being the official controller. They can cut spending in any area they choose but the budget has to balance.

    +

    http://citizenbudget.com/

    +

    Spending interactives

    +

    The World Bank have a spending database called Boost. It's a technical collaboration between the World Bank and various governments to digitize their spending data. It is especially targeted at countries that wouldn't otherwise have the technical resources. One of these countries is Kenya and the data has been displayed to great effect here:

    +

    +
    The examples given here for interactive infographics showing both budget and spending data are examples of where the user can navigate for themselves. Please let us know if you have examples of step-by-step guides to Budgets and Spending as it would be great to include them here.
    +

    Online reports

    +

    A report allows you to expand on points and show lots of different views of a dataset that would be just too cluttered on one infographic or in an interactive. A report is a good place for a step-by-step guide through a more extensive study.

    +

    An example of a very short report is from Enough is Enough (EiE), a coalition of Nigerians aiming to enhance citizen engagement and good governance. Their EiE report on the state of the Nation shows a very concise summary of the major spending corruption stories they identified that year, along with some links to sources.

    +

    Podcasts and videos

    +

    A well made podcast will almost certainly be shared widely given the right push. Describing the significance of your data and getting expert comments is so much easier to digest in a podcast than a long report - even if they are both very well written. Again, the technology to make radio has radically reduced in price in recent years.

    +

    The NPR Planet Money podcast rose to fame for its incredibly accessible, award-winning coverage of the 2008 Global Financial Crisis. Planet money now covers financial issues all over the world on its weekly podcast.

    +

    Online portal

    +

    A portal is suitable for a large and relatively frequently updated dataset. People in your community can then build web applications to pull the data. An example of a portal that might be used to add more context to spending data is the police.uk website. This is a complete view, updated monthly, of crimes committed in the UK, with details down to street level. The site also provides an API[15] which you can use to add new data to an app showing, for example, spending on police with crime levels overlaid on map.

    +

    Mobile technologies

    +

    There are a few free and open tools that will allow you to display your data through a mobile phone or tablet app. AKVO is a non-profit foundation and they describe (http://www.akvo.org/blog/?p=4822) how their AKVO FLOW (http://www.akvo.org/blog/?p=4836) platform works using free open source software to communicate your data through an Android mobile app.

    +

    Social media

    +

    Building a community around your cause in general and data analysis in particular is really valuable. Setting up a mailing list and creating a space to collaborate like on a wiki can lead to some really interesting social connections at relatively low cost can allow you to work with some really devoted and knowledgeable people. Some nice examples are the OKFN mailing lists (http://lists.okfn.org/mailman/listinfo) using Mail Man - http://www.gnu.org/software/mailman/index.html), wikis (for example http://www.wikispaces.com/).

    +

    There are also the big social media companies like Twitter and Facebook of course! These are great for reaching large numbers of people who have anything from a passing interest to complete devotion to your cause. When you join create a Twitter account or Facebook page with the name of your organisation. The strategy when joining these is to contact some high profile tweeters or facebookers to take on your cause or message. Make sure there are plenty of links back to your website when you Tweet or post to Facebook or other platforms. If you have a data release then create a package of the data, the story and any other resources (image downloads, reports etc) on a webpage and then tweet a link out with a link to that page. The same principle applies with Facebook.

    +

    Communicating with the press

    +

    When writing a press release keep it simple and very relevant to a story the journalist might like to run - leaving descriptions of your organisation or how you got the data to the very end or ideally include only a link to a relevant webpage. To be relevant you need to hook into something in the news cycle (http://en.wikipedia.org/wiki/24-hour_news_cycle) or offer up such a strong story it would create news. If it is the latter, then you might want to give it to a number of newspapers with an embargo so they will release the story all at the same time to create a really big impact.

    +

    If you are looking for news hooks then at the simplest level the data can relate to the time of year, maybe when there is a new financial phase or an election and the news is focused very much on spending and budgets. For example if it's a complete cleaned up version of all public spending, or budget or all the pension funds, just showing this data qualifies as something a journalist would find interesting - they may even have an in-house team working on the same thing so will be pleased to have the data quicker. Here the press release can be quite simply describing the data if the data is detailed and relevant enough to stand alone, along with a link to your portal and some instructions for using it, or a download of the data.

    +

    If you're wondering about how much extra work you need to do to present your data before sending out a press release then the same principles apply as when adding content to your website. If you have a great story from a great piece of analysis or just great data alone, then send the data and the story in a concise form with links through to the methodology on your website.

    +

    One final tip before describing what you might include in your press release: when working with journalists be sure to get a link to your website included in the piece.

    +

    Including images

    +

    If you do have a dataset with a story that would really benefit from an infographic then add it to the press release. Or if you have a mobile or tablet app, a podcast or report then add a screenshot of the most enticing view of it too. Ideally attach a small copy of the image to the email and give a link to a higher resolution version on your website.

    +

    It's likely the newspaper's graphics team will create a version in the newspaper's style so you should include the data for your graphic in a usable format (no PDFs).

    +

    The journalist may also choose to use your graphic as it is, so be sure to add a small discrete copy of your logo in the corner of the image or other citation embedded in the image.

    +

    Including an interactive infographic

    +

    Equally you may have made a descriptive interactive, in which case you might want to make it 'embeddable' on other websites. You may also ask for the news site to link back your own website, or include a link in the embeddable portion of the infographic.

    +

    Communicating with governments

    +

    In modern democracies we would advise CSO's to influence government decision making with evidence-based, factual, easy to understand policy proposals. These proposals can be related to almost any particular subject, from energy efficiency, labour policy, immigration, minority issues to public finance. Government relation to these policy proposals can be decisive in times of elections, and for this reason they cannot ignore them. Government capacity is limited, so you might find it takes a while to get a response from them.

    +

    For CSOs it is important to differentiate between Government's general goals, and policy specific goals. For example, every responsible Government has a general goal of maintaining a balanced fiscal policy. As we get more specific about this goal, stakeholders' opinions begin to differ. Governments are rarely very clear about their policy goals, as this would make it easier for CSOs to keep the Government accountable. Real policy goals can also be very different from what Government declares as its policy goals. For example, every Government is against tax evasion, but in many cases Governments indirectly support them, one way or another.

    +

    Common themes for communicating

    +

    To summarize, the key themes are: have a great story, be clear, be connected to lots of other people and be accountable by publishing methodology and data. These apply equally if you are communicating with the public or the media.

    +

    Being clear is making easily repeated points with the data, or describing a juicy data set in one sentence. If you can make the presentation fun, fascinating, or just beautifully simple then you are on to a good thing.

    +

    Being connected is making things technologically very shareable and then facilitating the spread of your message through your own community of supporters and social media.

    +

    Being accountable is being completely transparent about how you obtained, cleaned and analysed the data. This method has to be explained very clearly too, and the standards maintained for every release until your CSO has a good reputation of trustworthiness. But please don't let concerns about trustworthiness stop your CSO from exploring and having fun with the data in the first place, which is really something you need to consider when settling on a data story.

    +

    The final points are wherever possible, have fun telling your story and don't be afraid to experiment with different styles and tones but don't delay if you have a great dataset on your hands - let the world know!

    Appendices
    11. Resources
    12. Glossary

    11. Resources

    +

    There are a variety of resources that you can use to accomplish some of the technical tasks described in the book. Some are geared towards users who are already experienced in a programming language, and some can be used by those with no experience. Most are helpful with analyzing and visualizing data.

    +

    For analysis and research

    +

    For visualisation and cleaning 

    +

    Free and Open Source:

    +

    Others

    +

    12. Glossary

    +

    aggregated

    +

    data that is summarized using statistical methods or by simply summing its component parts

    +

    API

    +

    Application Programming Interface: a specification allowing two pieces of software to interface with each other, without either having knowledge of the inner workings of the other

    +

    backlog

    +

    a list of tasks prepared by the project manager

    +

    bulk data

    +

    refers to the entirety of a dataset, instead of the incremental amounts that may be retrieval via other means, such as an API

    +

    CSO

    +

    Civil Service Organizations: a legally constituted organization created by natural or legal persons that operates independently from any form of government in the interest of civil society. Also commonly referred to as NGOs.

    +

    CSV

    +

    Comma Separated Values: A format for transactional data that separates each column value by a comma. Can be imported and exported from most spreadsheet programs

    +

    database query language

    +

    a syntax for interacting with data in a database. Usually specific to the software being used

    +

    hacker

    +

    one who programs enthusiastically (even obsessively) or who enjoys programming rather than just theorizing about programming (like a 'coder' - but more energetic :) )

    +

    hackday

    +

    a session where hackers come together to answer a particular task

    +

    iteration

    +

    a period of a defined length during which changes to a project can be made. Usually ends in a review meeting to plan the next iteration.

    +

    JSON 

    +

    Javascript Object Notation: a data format that is relatively easy for humans to read and write and very easy for machines to read and parse

    +

    Mirroring

    +

    creating an entire copy of a website on another server, primarily for backup

    +

    MySQL

    +

    (pronounced "My Sequel" or "MY S-Q-L"): Open source relational database software

    +

    nominatim

    +

    A web service which turns place names and addresses into coordinates so that they can be mapped / associated with an administrative area

    +

    PostgreSQL

    +

    (abbreviated Postgres): Open source object relational database software, a query language similar (but not identical) to that in MySQL is used

    +

    OCR

    +

    Optical Character Recognition: the electronic conversion of scanned images of handwritten, typewritten or printed text into machine-encoded text

    +

    reference data

    +

    data from outside an organisation (often from standards organisations) which is, apart from occasional revisions, static

    +

    scope creep

    +

    When a project's scope slowly and steadily increases, without a formal re-examination of the original terms and requirements

    +

    shapefile

    +

    a file format for geospatial vector data that is primarily used to feed data into mapping applications

    +

    Tilemill

    +

    Hosted and open source mapping software for creating maps

    +

    user story

    +

    short descriptions that outline in plain language the different use cases and possibilities for different types of users for the purposes of determining software requirements

    +

    vector data

    +

    a type of image composed of connected points rather than pixels

    +

    XML

    +

    eXtensible Markup Language: a markup language created to structure, store, and transport data by defining a set of rules for encoding documents in a format that is both human-readable and machine-readable.

    +

    Some of the above entries contain excerpts from Wikipedia.org and the Jargon File v. 4.4.8

    diff --git a/examples/openspending/content/resources/handbook/book-config.js b/examples/openspending/content/resources/handbook/book-config.js new file mode 100644 index 00000000..ae8bfb52 --- /dev/null +++ b/examples/openspending/content/resources/handbook/book-config.js @@ -0,0 +1,19 @@ +paginationConfig = { + 'sectionStartMarker': '.objavi-subsection', + 'sectionTitleMarker': '.objavi-subsection-heading', + 'chapterStartMarker': 'h1', + 'chapterTitleMarker': 'h1', + 'flowElement': "document.getElementById('book')", + 'alwaysEven': false, + 'columns': 1, + 'enableFrontmatter': true, + 'bulkPagesToAdd': 50, + 'pagesToAddIncrementRatio': 1.4, + 'frontmatterContents': '
    Spending Data Handbook
    ' + + '
    ' + + 'Iteration 1
    ' + + '
    Copyright 2012: Lucy Chambers (OKFN), Lisa Evans (OKFN), Adam Hyde (Booksprints.net), Kaitlin Lee (Sunlight Foundation), Friedrich Lindenberg (OKFN), Damir Mehmedbasic (Public Interest Advocacy Center), Sona Mitra (CBGA), Federico Ramirez (Fundar), Sam Smith (OKFN).
    License: CC-BY
    ' + + '
    ', + 'autoStart': true, +} + diff --git a/examples/openspending/content/resources/handbook/ch001_introduction.html b/examples/openspending/content/resources/handbook/ch001_introduction.html new file mode 100644 index 00000000..59c58afc --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch001_introduction.html @@ -0,0 +1,90 @@ +--- +title: Introduction +layout: handbook +--- + +
    +
    + +
    +
    +

    Introduction

    +

    The Spending Data Handbook is addressed to people and organisations who want to use and understand government budgets and spending data in their work. These groups may include government oversight and accountability groups, issue oriented advocacy organisations, journalists covering the latest round of budget cuts or austerity measures, or citizens who just want to inform themselves on what their government is spending. The truth is that for many countries, it's never been easier to access the underlying data relating to government budgets and spending and the expectations have never been higher for governments to release this information in a timely manner. But once you've got the information, making sense of it can be daunting.

    +

    Irrespective of which kind of user is approaching this data, they all have one thing in common: they need to manipulate, analyse and interpret data in their work. Increasingly, we are bombarded with statistics and figures from governments and political parties and it is the job of these organisations to scrutinise and fact-check them, as well as come up with alternative models and solutions. This requires timely, data driven analysis.

    +

    If these groups think that "topic-driven analysis" can drive positive change within governments, there is another group at whom this book is addressed: a strange race of people who self-identify as "civic hackers" and think that technology can be put to work to make government as easy, accessible and effortless for citizens as the internet. While these groups use data processing tools in their day to day lives, their focus is often on enabling others to act on information, rather than having a particular advocacy aim themselves (though there are a few notable exceptions to this rule).

    +
    DON'T PANIC! This book uses a bit of technical vocabulary and terms that may be unfamiliar or used differently in different circles. Anything in italics such as hacker in this paragraph, can be found in the glossary at the back, it may not mean what you think.
    +

    The range of topics tackled by these groups is so diverse that it would be impossible to address all of the questions and issues in one humble book. However, there are some overarching principles and helpful techniques which apply universally to working with government financial data and particularly, how technology might help to do so. This book aims to highlight areas where civic hackers, citizens, Civil Society Organisations (henceforth CSOs), and civil servants working in the field of government transparency could combine forces to achieve common aims. Like the Open Data Handbook (http://opendatahandbook.org/en/), it will be available as a continually evolving, open, educational resource on the internet.

    +

    What we will cover in this book

    +
    • Collaborating with other organizations to pool resources and strengthen your advocacy effort
      • +
    • If you're just starting out, what data to look for and what to ask for (nay, demand!) from your government
      • +
    • The 'Data Pipeline': Tricks and tips for finding, wrangling and systematically processing your data
      • +
    • Getting ambitious, running a technology project
      • +
    • Presenting your findings to engage the public, media and government
      • +
    • Lists and appendices of technical and non-technical resources
      • +

    The Data Processing Pipeline

    +
    +

    How the book was created

    +

    The book was started at a four day book sprint bringing together organisations from around the world from both a technical and a CSO background. Representatives from the Open Knowledge Foundation (UK), Fundar (Mexico), the Centre for Budget and Governance Accountability (India), the Public Interest Advocacy Centre (Bosnia and Herzegovina), and the Sunlight Foundation (US) contributed to the original document. The Book Sprint was facilitated by Adam Hyde, founder of the Book Sprint method and www.booksprints.net.

    +

    Why are we writing this?

    +

    As people who work with this data, we know better than anyone that a few years of a handful groups spreading awareness of spending data to the general public has not been sufficient for enacting earth-shattering change in the budgetary policies and processes in governments around the world. While it's true there's more spending data available than ever before, these efforts have proven to be superficial in some cases and the budget processes themselves have not become much more transparent or participatory.

    +

    Conversely, the armchair auditors that were expected to emerge from the citizenry never really materialised. The notion of citizen auditors and engaging with citizens on an individual and collective basis still struggles to make headway. Clearly, we need to take solutions to this problem to the next level. More groups need to be more involved than ever before.

    +

    What makes this work so tricky?

    +
    • The data people need is not available to the public (and to CSOs).
      • +
    • Alternatively, the data is not available until after all the important decisions within government have been made.
      • +
    • It's difficult to simplify data as complex as budgets and spending and make it accessible to a variety of audiences.
      • +
    • The discourse on budgets and governance in the country is usually replete with jargon and technicalities.
      • +
    • Even when compelling research findings are presented, there's no sense of urgency on the part of key policy actors. Advocacy with a range of stakeholders becomes important.
      • +
    • There is duplication of efforts. One CSO may invest two weeks of its resources into painstakingly cleaning up and extracting data from a policy document published as a PDF while another will be doing exactly the same thing.
      • +
    • Organizations doing this work often lack any kind of peer review process among similar groups.
      • +
    • There may be skills gaps at many CSOs. Skill and knowledge sharing can help all of them achieve common goals.
      • +

    Contribute to the book

    +

    This book is released under a Creative Commons attribution licence, meaning that anyone is free to use and reuse the material provided that it is attributed to the Spending Data Handbook. You can help by doing one of the following:

    +
    • Translate it and customise it for your region. The examples which we include here are the ones which we know best, however, you may feel that in your area, there are topics or examples which would be more relevant. Take the book, remix it and add your own examples.
      • +
    • Correct it and update it - treat it like a Wikipedia article, a living document. The only way that this book will stay relevant and factually correct is with the help of you and other people who know your stuff.
      • +
    • Be inspired - we are filling the book with lots of examples of visualisation and data so inspire you and people you work with.
      • +
    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch002_working-with-others.html b/examples/openspending/content/resources/handbook/ch002_working-with-others.html new file mode 100644 index 00000000..19499732 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch002_working-with-others.html @@ -0,0 +1,64 @@ +--- +title: Working with others +layout: handbook +--- + +
    +
    + +
    +
    +

    Working with others

    +

    We've outlined some ambitious goals for this book, but you don't have to go it alone! In this section we focus on how to work with other organizations to create powerful advocacy efforts, all driven by hard fiscal data! Every CSO, not just government transparency organizations, can leverage the information in this book. Government spending data touches issues covered by a variety of organizations and journalistic outlets, such as human rights, the environment, poverty, scientific research, education, economic policy, and more. But not all these groups may realize it's importance.

    +

    Why work together?

    +

    A common observation has been that CSOs, journalists, and other groups that work with a lot of data encounter a set of similar problems. The problems typically include lack of technical knowledge, duplication of work, lack of wide dissemination of analysis and results, and creating compelling visual products. A lot of these challenges could be mollified if proper channels of communication existed between these organizations.

    +

    One of the most important ways to reduce these technical barriers is to maintain an active dialogue and a process of sharing the methodologies and underlying data in analysis you produce. You can avoid the mistakes of your predecessors by reading their methodologies and learning about idiosyncrasies in the data. Or you may be able to point out mistakes another organization has made. Similarly, if you document how your data has been processed and analyzed, other organizations can benefit (more about that in a few chapters).

    +

    Creating communication can be as easy as joining or maintaining a mailing list of groups that work on similar topics or leverage similar data in their work products. Most webmail services offer free group or mailing list functions, such as Google Groups or Yahoo Groups. If you maintain one, be responsive to messages and active in recruiting members. Email is an extremely low barrier for starting a conversation between organizations.

    +

    If you've already got data products that you want to share, try disseminating them in a couple of different ways. In addition to the methods outlined above, try preparing primers or manuals for other groups that want to start working with datasets that you may be particularly experienced with. You can also organize workshops, meetups, and webinars to familiarize groups with the data you work with. If grassroots issue groups can internalize your data products and present them in a contextually relevant way to their user base, then both groups have gained something from the partnership.

    +

    Another good way to facilitate knowledge sharing and communication between organizations is by conducting internship programmes for interested individuals and organisations. However it is important to keep in mind the target groups. It's great to interface with similar spending data focused groups working at different levels of government than your own, but identifying groups that work on completely different issue areas is important. These groups may feel incapacitated to enact change in the fiscal policy as it relates to their own issues. It then becomes an important job for spending data groups to identify and offer their support. It's especially important to establish strong links with groups that serve those minority populations who are often marginalized in government spending priorities.

    +

    The kind of collaboration discussed above can result in a much broader and stronger coalition of organizations that can advocate for fiscal transparency and detailed disclosure of spending data at all levels of government.

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch003_gov-gov-collaboration.html b/examples/openspending/content/resources/handbook/ch003_gov-gov-collaboration.html new file mode 100644 index 00000000..ebd6a5be --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch003_gov-gov-collaboration.html @@ -0,0 +1,85 @@ +--- +title: Helping the government help itself +layout: handbook +--- + +
    +
    + +
    +
    +

    Helping the government help itself

    +

    We've discussed that it's important for CSOs, journalists, and other groups to work together and it's likely that you won't bump up against too much resistance to this idea. But what about when you get pushback from the government even after you've built a strong coalition to advocate for better access to data? Well, one answer you can give them is "Government darling, you're only hurting yourself".

    +

    Whenever the word 'transparency' is mentioned, the first thought that usually springs to mind is a bright light being shone into dark corners of government offices, trying to expose the dark secrets of corrupt bureaucrats and inefficient expenditure. What is often overlooked is that governments also often stand to benefit from more transparent publishing practices. Not only does proactive disclosure help save on the time and cost associated with increased Freedom of Information requests, but other governments within the same nation can benefit from each other as well.

    +

    Connecting different levels of government

    +

    Money at the federal level is often transferred to local governments in the form of grants, direct payments or formula payments. When sub-national governments have better, more timely information on national budgeting priorities, it allows them to adjust their own budgets to account for cuts or increases in certain programs. Similarly, many local governments have small staffs and rely on revenue estimates and models at the federal level to estimate their own revenue.

    +

    This underscores the need for a consistent, standardized information sharing across governments, as it would allow them to share models, best practices, and software with each other, instead of custom building everything from scratch. Furthermore, it would allow for cross-checks on both the federal side and local side for data auditing purposes.

    +

    The next section highlights a couple of case studies for CSOs to use when trying to convince their government to publish better data.

    +

    Case study

    +

    Transparency to combat lag-times for Governments in British Columbia

    +

    When the province of British Columbia built a data portal (http://www.data.gov.bc.ca/), its motivations were primarily:

    +
    • citizen engagement - they wanted citizens to better understand the workings of government
      • +
    • innovation - they wanted people to build applications and tools using the data
      • +
    • making handovers effective - a large number of the workforce were approaching retirement age, and those in charge wanted to make sure that they handed over the necessary information well in advance
      • +

    Probably one of the less-expected impacts was seeing how civil servants themselves used the portal. In 2012 approximately one third of all the traffic originated from government computers. The technology enabled faster access to relevant data within the government departments, contributing to better collaboration on policies that required fiscal data. There was also about 20% increase in the number of Freedom of Information (FoI) requests, showing that releasing a small amount of data fuelled wider interest in data.

    +

    The importance of constant access

    +

    It is somewhat terrifying to think that a lot of budgeting decisions could made almost entirely in the dark. As soon as one executive budget proposal is finalised and published, work often begins on producing the next one. Within governments, those who have to draw up the next year's plan need access (and quickly) to information, such as actual quarterly expenditures, in order to work out whether a government department is properly resourced from the outset, or is drastically under or over spending.

    +

    Case study

    +

    International Aid Transparency Initiative (IATI)

    +

    Building on around 20 years of previous work, the first version of the International Aid Transparency Initiative (IATI) standard was launched in early 2011 and aid donors around the world started publishing to it. The IATI specification documents data about both aid donors and aid activities, enabling comparison and encouraging good practice in data management. The IATI standard also includes space to attach relevant documents and details of project results, to allow the standard to build context around the data and become an end-to-end solution, tracking projects from inception to execution.

    +

    The IATI standard has seen far greater uptake than any previous effort in terms of organisations electing to publish their data in a standard format. These successes are often attributed to the multi-stakeholder nature of the design process, including both policy makers and technical experts. From 2009 to 2010, IATI consulted with a wide range of stakeholders on the design of the technical standard, alongside a parallel process to secure donor support for publishing their aid information.

    +

    In the past it was common to respond to information shortages by building a new database. But by working with open data principles, IATI allows a more distributed solution - where information can flow between organisations in many different ways, not just into a central database.

    +

    Main user groups

    +
    1. Parliamentarians in developing countries gain a better oversight of the aid resources available. Knowing where to allocate resources in their own budgeting processes is vital to ensure that money is spent in the best way. Sometimes, there is transparency-asymmetry between different parts of government. For example, a treasury may be very willing to open up the information it holds, but departments which benefit strongly from aid donations (e.g. departments of health) may be more reluctant to be transparent about aid revenues, as they will not want to 'lose out' from central government budgeting.
      2. +
    2. Donors: Know where their money is going and whether it is being spent correctly.
      3. +
    3. CSOs and private companies who monitor aid effectiveness
      4. +

    The benefits of this approach have already been demonstrated, with many CSOs and charities choosing to follow the IATI Standard, although it was developed primarily with governments in mind.

    +

    Donors publish aid information as a feed which can be read by many different applications, including those created by other donors, by the open data community, and - importantly - by software providers that are developing country financial systems. By providing aid information in a standard format, many different users can access the data in the way they need to - and developing countries can see the resources, which are supposed to be flowing to them.

    +

    Is there anything like this for spending and budgets?

    +

    At present, no. The way many governments report their financial data is a function of how their budget process works or how they use their accounting software. This may not even be consistent within a government, but vary by department and sub-department.

    +

    Could it be useful to create one? We suspect so. The IATI standard for the first time is enabling people to track the money across country boundaries, something which is almost impossible with other types of expenditure, but crucial for those whose job is oversight of government activities and accountability bodies, such as those who work to prevent international money laundering and corruption.

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch005_introduction-to-data-literacy.html b/examples/openspending/content/resources/handbook/ch005_introduction-to-data-literacy.html new file mode 100644 index 00000000..7b3eada7 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch005_introduction-to-data-literacy.html @@ -0,0 +1,89 @@ +--- +title: Data-driven advocacy and research +layout: handbook +--- + +
    +
    + +
    +
    +

    Data-driven advocacy and research

    +

    We are now in a phase where many governments around the world are proactively publishing documents about what they plan to spend (budgets) and actually spend (spending data). Increasingly, this material is available on the internet, so that anybody can access it at any time. Still, too much of the information is released in the form of 'documents' rather than 'data'. Ideally we need both so that inforamtion can be analyzed, re-used and understood. This chapter is a quick overview of some of the raw inputs required for data-driven advocacy and how it works in practice. 

    +

    What do we mean by machine-readable data?

    +

    When we speak about data, what we usually refer to is the notion of machine-readable (http://en.wikipedia.org/wiki/Machine-readable_data) data. Many of the formats most commonly used for policy papers and long-form reports published by most policy-making institutions are PDF files, Word documents, web pages or closed interactive infographics - do not structure information in a way that lends itself to automated analysis and extraction.

    +

    Such documents are formatted for humans (or printers) to interpret, and it can be hard (and in many cases nearly impossible) for a machine to re-construct the elements in the presentation.

    +

    Other formats, such as Excel and CSV files contain a higher level of structured information. For example, in an Excel file you can mark a number of cells and easily calculate their sum. Even more exotic and useful file formats, such as XML documents, JSON APIs or Shapefiles may not have easy-to-use viewer applications. You can think of them as the glue that connects different systems on the web, so that different databases can work together in a seamless fashion.

    +

    Why do CSOs need it?

    +

    What asking for machine-readable bulk data means for CSOs is simple: you won't have to spend a lot of time manually extracting data from reports into spreadsheets to be able to filter, sort and analyse it - a process which is both time-consuming and can introduce errors.

    +

    What to ask for when asking for data: a checklist

    +

    In the next section 'Getting Data' - we will deal with asking governments for data (or getting it via other means). To set the scene for this and to work out whether your government actually publishes usable data already, have a quick look at the following questions:

    +
    • Is the government's data published in a machine-readable format? E.g. CSV, XML, JSON. While there is nothing wrong with publishing a PDF to support a data release (in fact it is often nice to have a nicely-laid out document to cross reference and sanity-check data) it shouldn't be the only thing published and if you are asking for a policy document, ask for the underlying data in a spreadsheet so you can check the numbers.
      • +
    • Does the government publish a 'data dictionary' to explain the terms used in the dataset? This should include definitions of column headers, explanations of terms and ranges used within the main body of the data, explanations of any changes in terminology which have been introduced since last time the dataset was released
      • +
    • How is the data that is being published actually used internally by governments? Do some sanity checks on the minimum and maximum values of different columns to make sure they fall into the documented ranges and don't seem out of place. Do you see negative values when you don't think you should? Negative values usually mean money owed.
      • +
    • Is the structure of the data the same across years? If not is there a description of how it changes? It never hurts to contact the publisher and ask questions about the change and why it occurred. The publisher may have their name and contact details on the report or webpage. If there is no named contact then call the department's enquires number or send a message to their email address asking to meet or discuss the data.
      • +
    • How aggregated is the data? What is the number of real-world financial transactions that are expressed by a single line of the dataset you have? For budgets this will mostly be hard to tell - but with transactional expenditure you want to make sure that the data is fairly disaggregated. Ideally, each entry represents a transaction - but even if this isn't true you'll still want to ensure the number is not in the tens or hundreds of thousands (e.g. government programmes as a whole).
      • +
    • Ask for reference data. If your budget or spending data is augmented with reference data, make sure you have access to it. This might include functional or category codes on budget line items, location codes for describing recipient location, or codes that indicate the status of the record. 
      • +
    • Ask also for the guidelines people were given when creating the dataset. This will make it easier to understand what is included within the data, e.g. are the numbers in thousands / millions. 
      • +
    • Final tip: if the data you want is not given then narrow your scope. Your chances of success will be higher if you narrow the scope of the data you're requesting from the government and you are specific. Government is the de facto keeper of all kinds of data, so parameters that narrow your request are always helpful.
      • +

    An introduction to data-driven advocacy

    +

    Is going out and provoking a riot the best way to get a Government to take onboard your message? There are alternatives: hit them with the data hammer instead!

    +

    Making evidence-based policy proposals consists from three major phases: formulating your assumption, analysis (which often leads to re-formulating your assumption, and presenting your data in an engaging way in a policy proposal.

    +

    Analysing assumptions

    +

    Asking the right question is key to getting the most out of your data. We all make assumptions, and our organisation may have a particular standpoint on a given issue. Our first task is always to formulate our assumptions and then interrogate them ferociously. Although we try to be rational in this process, our judgement is often influenced by our subjective goals, values, and beliefs. Sometimes, you'll need to revisit your assumptions several times over to ensure they are valid and you can back them up with data. Once you know your policy problem is definitely a problem, you can work to package it in a way that's appropriate for your target audience. 

    +

    What is public interest? 

    +

    Often our job is to act in the public interest by analysing conflicting assumptions and working out which one is more valid. For example, in Greece, Spain, and many other European countries people protest almost everyday as the Government cuts spending to bring down its budget deficit. If the Goverment wanted to keep its current level of spending, but increased taxes to increase its revenue, different citizens groups would still protest depending on which taxes are to be increased. In any case, there will always be more than one interpretation of any Government policy, and interested side to support it, or not.

    +

    Policy analysis

    +

    Once we have a well defined policy problem, specific goals, or results different stakeholders are trying to achieve, and corresponding instruments they are using in this process, we may systematically search for the specific data needed to create our own policy proposals. This data can be obtained either from the Government, some other sources e.g. academic journals, private companies, or generated by ourselves. When data is gathered we will use a specific methodology to analyze it, and based on this analysis we will approve or reject our assumptions. If the assumption is rejected, based on our findings we will have to make the new assumption, and start the process from the beginning. If our assumption is approved, we will use our results to make a policy proposal to the Government.

    +

    Policy proposals

    +

    For CSOs it is important to recognize who is a decision maker, hence, who you should be targeting with your policy proposal. Policy proposals should be methodologically well structured, evidence-based, open for debate, and scientificaly evaluated. Governments will seldom take our policy proposals as their own policymaking, but may actually change its course of action, get new insights, views, and understanding of the subject. We may also use policy briefs to approach Government officials, or press releases to get the attention of the public. 

    +

    +

    Case study 

    +

    Fish subsidies

    +

    The influence CSOs have on government policy comes from a wide and varied set of activities. These can range from producing a widely shared dataset or infographic which subtly influences the mood of policy makers, to more targeted CSO advocacy and lobbying on issues they are experts.

    +

    The Fish Subsidies group (http://fishsubsidy.org) are a nice example of a CSO engaged in targeted activites. Having collected a comprehensive set of data on Fishing subsidies paid under the European Union’s common fisheries policy and they break this down into payments for every EU member state, and then complemented this with activites of fishing. They have produced a report (http://is.gd/XYPgq5) assessing the environmental and social impacts of the Financial Instrument for Fisheries Guidance between 2000 and 2006. This extensive document fed directly into the EU political decision making process.  

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch006_types-of-data.html b/examples/openspending/content/resources/handbook/ch006_types-of-data.html new file mode 100644 index 00000000..8c89ff1a --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch006_types-of-data.html @@ -0,0 +1,91 @@ +--- +title: Types of data +layout: handbook +--- + +
    +
    + +
    +
    +

    Types of data

    +

    For the purposes of this manual, we have decided to abstract the meaning of the terms "budget data" and "expenditure data" to fit into a broader context. While these terms may have different meanings on a country by country basis, they are intended to be taken as defined in this section, throughout the rest of the manual. In this section, we look briefly at the two different types of data and what questions can be addressed using them.

    +

    +

    Budget data is defined as data relating to the broad funding priorities set forth by a government, often highly aggregated or grouped by goals at a particular agency or ministry. For instance, a government may pass a budget which contains elements such as "Allocate $20 million in funding for clean energy grants" or "Allocate $5 billion for space exploration on Mars". These data are often produced by a parliament or legislature, on an annual or semi-annual basis.

    +

    Spending data is defined as data relating to the specific expenditure of funds from the government. This may take the form of a contract, loan, refundable tax credit, pension fund payments, or payments from other retirement assistance programs and government medical insurance programs. In the context of our previous examples, spending data examples might be a $5,000 grant to Johnson's Wind Farm for providing renewable wind energy, or a contract for $750,000 to Boeing to build Mars rover component parts. Spending data is often transactional in nature, specifying a recipient, amount, and funding agency or ministry. Sometimes, when the payments are to individuals or there are privacy concerns, the data are aggregated by geographic location or fiscal year.

    +

    The fiscal data of some governments may blur the lines of these definitions, but the aim is to separate the political documents from the raw output of government activity. It will always be an ultimate goal to link these two datasets, and to allow the public to see if the funding priorities set by one part of the government are being carried out by another part, but this is often impractical in larger governments since definitions of programs and goals can be "fuzzy" and vary from year to year.

    +

    +

    Budget data

    +

    Using the definitions above, budget data is often comprised of two main portions: revenue and taxation data and planned expenditures. Revenue and spending are two sides of the same coin and thus deserve to be jointly considered when budget data is released by a government. Especially since revenue tends to be aggregated to protect the privacy of individual taxpayers, it makes more sense to view it alongside the budget data. It often appears aggregated by income bracket (for personal taxes) or by industrial classification (for corporate taxes) but does not appear at all in spending data. Therefore, budget data ends up being the only source for determining trends and changes in revenue data.

    +

    Somewhat non-intuitively, revenue data itself can include expenditures as well. When a particular entity or economic behaviour would normally be taxed but an exception is written into the law, this is often referred to as a tax expenditure. Tax expenditures are often reported separately from the budget, often in different documents or at a different time. This often stems from the fact that they are released by separate bodies, such as executive agencies or ministries that are responsible for taxation, instead of the legislature (http://internationalbudget.org/wp-content/uploads/Looking-Beyond-the-Budget-2-Tax-Expenditures.pdf).

    +

    Budgets as datasets

    +

    A growing number of governments make their budget expenditure data available as machine-readable spreadsheets. This is the preferred method for many users, as it is accessible and requires few software skills to get started. Other countries release longer reports that discuss budget priorities as a narrative. Some countries do something in between where they release reports that contain tables, but that are published in PDF and other formats from which the data is difficult to extract.

    +

    On the revenue side, the picture is considerably bleaker, as many governments are still entrenched in the mindset of releasing revenue estimates as large reports that are mostly narrative with little easily extractable data. Tax expenditure reports often suffer from these same problems.

    +

    Still, some areas that relate to government revenue are beginning to be much better documented and databases are beginning to be established. This includes budget support through development aid, for which data is published under the IATI (http://www.aidtransparency.net/) and OECD DAC CRS (http://stats.oecd.org/Index.aspx?DatasetCode=CRSNEW) schemes. Data about revenues from extractive industries is starting to be covered under the EITI (http://eiti.org/) with the US and various other regions introducing new rules for mandatory and granular disclosure of extractives revenue. Data regarding loans and debt is fairly scattered, with the World Bank providing a positive example (https://finances.worldbank.org/), while other major lenders (such as the IMF) only report highly aggregated figures. An overview of related data sources can be found at the Public Debt Management Network (http://www.publicdebtnet.org/public/Statistics/).

    +

    Connecting revenues and spending

    +

    It is highly desirable to be able to determine the flow of money from revenues to spending. For the most part, many taxes go into a general fund and many expenditures come out of that general fund, making this comparison moot. But in some cases, in many countries, there are taxes on certain behaviours that are used to fund specific items.

    +

    For example, a car registration fee might be used to fund the construction of roads and highways. This would be an example of a user fee, where the main users of the government service are funding it directly. Or you might have a tax on cigarettes and alcohol that funds healthcare grants. In this case, the tax is being used to offset the added healthcare expense of individuals taking part in at-risk activities. Allowing citizens to view what activities are taxed in order to pay for other expenditures makes it possible to see when a particular activity is being cross-subsidized or heavily funded by non-beneficiaries. It can also allow them to see when funds are being diverted or misused. This may not always be practical at the country level, as federal governments tend to make much larger use of the general fund than other local governments. Typically, local governments are more comprehensive with regards to releasing budget data by fund. Having granular, fund-level data is what makes this kind of comparison and oversight possible.

    +

    +

    What questions can be answered using budget data?

    +

    Budget expenditure data has an array of different applications, but it's prime role is to communicate to it's user broad trends and priorities in government spending. While it can help to have a prose accompaniment, the data itself promotes a more clear-cut interpretation of proposed government spending over political rhetoric. Additionally, it is much easier to communicate budget priorities by economic sector or category than it is at the spending data level. These data also help citizens and CSOs track government spending year over year, provided that the classification of the budget expenditure data stays relatively consistent.

    +

    Spending data

    +

    For most purposes, spending data can be interpreted as transactional or near-transactional data. Rather than communicating the broad spending priorities of the government like budget data should, spending data is intended to convey specific recipients, geographic locations of spending, more detailed categorization, or even spending by account number.

    +

    Spending data is often created at the executive level, as opposed to legislative, and should be more frequently reported than budget data. It can include many different types of expenditures, such as contracts, grants, loan payments, direct payments for income assistance and maintenance, pension payments, employee salaries and benefits, intergovernmental transfers, insurance payments, and more.

    +

    Some types of spending data - such as contracts and grants - can be connected to related procurement information (such as the tender documents and contracts) to add more context regarding the individual payments and to get a clearer picture of the goods and services covered under these transactions.

    +

    Opening the checkbook

    +

    In the past five years, there have been a spate of countries and local governments that have opened up spending data, often referred to as "checkbook level" data. These countries include, but are not limited to, the US (including various state governments), UK, Brazil, India (including some state governments) and many funds of the European Union.

    +

    Disclosure thresholds

    +

    At least two of these countries have imposed seemingly arbitrary thresholds on the size of transactions that are included. For example, the US and the UK exclude transactions under $25,000 and 25,000 GBP, respectively. Are these thresholds appropriate? That can't be known for sure without more information about how these numbers were arrived at. Principally, having thresholds or exceptions to the reporting of this data depends on the underlying systems that drive disclosure of this data. Are these systems linked directly with the accounting systems already used in the government, easing the burden of disclosure? If so, the threshold for excluding transactions should be very low (setting aside for a moment the cases that require redaction for privacy purposes).

    +

    If the systems are mostly divorced, as is the case with the US, then it begs the question, why? The more steps and processes between the internal government accounting systems and the public accounting systems, the higher the chance of error in the data and chance for omission of data. It also undermines the primary goal of public oversight if there are separate systems. However, governments often struggle with IT resources and contracting, which presents a tension between releasing any spending data at all, and release that is consistent with the above principles. If a threshold is necessary, then the amount should be consistent in size and scope with the overall expenditure level for that particular government. It is not appropriate, for example, that the threshold for the US State of Maryland spending reporting is also $25,000, when their annual budget is only a fraction of the federal government's budget.

    +

    Release early, release often

    +

    Spending data should be released in a relatively timely fashion, at least a monthly or quarterly basis. The timeliness of this data is what allows users to see if the spending priorities in the budget data are being reflected in the spending data. Also it allows the public and government stakeholders to view the current year's spending on a more detailed level as the next year's budget is being decided.

    +

    A good example of such release of spending information can be the Indian experience, especially the Employment Guarantee Programme, one of the major National flagship programmes on providing demand-based employment to the rural working age-group population in India. Its Management Information system (MIS) has become the most effective way of getting information on spending on a monthly basis. The data is updated monthly in an accessible spreadsheet format (Excel) at the sub-national government level. This makes the data transparent and available in the public domain to be equally accessed by all. The village level household database has internal checks for ensuring consistency and conformity to normative processes. It includes separate pages for approximately 250,000 local governments at the village level, 6,465 Blocks, 619 Districts and 34 States & Union Territories. The portal places complete transaction level data in the public domain.

    +

    However, problems related to maintaining an MIS in every state in a functional form and releasing continuous flow of data have been contentious issues. The major concerns emerge from lack in technical capacity as well as cost related issues. A cumbersome back-end system for supplying the data requires installation of a specific software with prerequisite configurations and technical operators with specific capacities. These requirements have raised costs and put a great demand on technology to ensure a continuous flow of data on the programme, specifically in most interior parts of the country and hence affect timely release of data.

    +

    What questions can be answered using spending data?

    +

    Spending data can be used in several different areas: oversight and accountability, strategic resource deployment by local governments and charities, and economic research. However, it is first and foremost a primary right of citizens to view detailed information about how their tax dollars are spent. Tracking who gets the money and how it's used is how citizens can detect preferential treatment to certain recipients that may be illegal, or if certain political districts might be getting more than their fair share.

    +

    It can also help local governments and charities respond to areas of social need without duplicating federal spending that is already occurring in a certain district or going to a particular organization. Lastly, businesses can see where the government is making infrastructure improvements and investments and use that criteria when selecting future sites of business locations. These are only a few examples of the potential uses of spending data. It's no coincidence that it has ended up in a variety of commercial and non-commercial software products -- it has a real, economic value as well as an intangible value as a societal good and anti-corruption measure.

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch007_getting-cleaning.html b/examples/openspending/content/resources/handbook/ch007_getting-cleaning.html new file mode 100644 index 00000000..48bda5ed --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch007_getting-cleaning.html @@ -0,0 +1,140 @@ +--- +title: Getting and cleaning data +layout: handbook +--- + +
    +
    + +
    +
    +

    Getting and cleaning data

    +

    As you look into a new, data-driven project, you'll find it easy to imagine all the analysis and comparisons that could be done to prove a particular point or to explore some area of government activity. As you then look into the availability of data on a particular topic, disillusionment will quickly set in: information may be hard to find, unstructured or just not available to the public at all. As you search for data, there are a number of possible techniques you may want to investigate. Some are based on the governments cooperation while others use technology to bridge the gap between how information is shared and what is necessary for a data-driven advocacy project.

    +

    Getting data

    +
    <p >Data processing pipeline
    +

    An increasing number of governments have accepted the need for pro-active transparency. When releasing open data, governments commit to making machine-readable information available on the web, in a form that enables anyone to use, re-use and re-distribute the data without legal or technical restrictions. Such releases should happen in a timely manner and include detailed, disaggregated data. Many countries have set up dedicated open data portals which provide easy access to all published datasets and relevant metadata (i.e. information about when a dataset was last updated, who published it and where documentation regarding format and contents can be found). The benefit of using open data when it's available includes the ease of access, but also usually a more authoritative base for any further analysis.

    +

    In some cases, open data is provided through an application programming interface (API), often a web-based method for retrieving, searching or even updating the available information dynamically. APIs provide up-to-date data in a granular and filtered form, removing the need to repeatedly process and update source files.

    +

    A common use case for APIs is relatively time-sensitive information, such as procurement calls and contracts which are released every day. In the UK, BusinessLink (http://www.contractsfinder.businesslink.gov.uk/data-feed.aspx) provides a number of data feeds which contain information about procurement notices. Similarly, the USASpending portal provides a set of APIs that can be used to retrieve up-to-date grants information for the US federal government (http://usaspending.gov/data).

    +
    If you are looking for a list of data catalogs from around the world, DataCatalogs.org is curated by experts in this area from around the world. Search for your country and see what you find.
    +
    +

    Freedom of information

    +

    Even before the rise of open data, many countries decided to increase the transparency of their governments by introducing freedom of information (FoI) legislation. Such laws enable every citizen to request documents and other material from parts of the government which do not merit special protection (e.g. due to concerns over privacy, national security or commercial confidentiality).

    +

    Journalists, activists and CSOs have long had channels of acquiring information. Sometimes, having a good relationship with a press-officer or a civil servant is good enough and making a formal request for information is unnecessary (your friendly press-officer may even feel slightly offended if you don't ask them nicely first). FoIs generate a lot of paperwork (hence grumpy civil servants), so if you do have the contacts, it may be a good idea to ask nicely first!

    +

    Freedom of Information requests often require some degree of preparation, so that the documents or databases that are requested are clearly identified, you know which department or unit is in charge of it and you can address possible concerns over privacy or commercial confidentiality in your request.

    +

    While freedom of information legislation is in force in many countries, it was often made before the need for structured data became apparent - thus many laws do not allow the citizen to specify a particular format. Many governments choose to release information on paper rather than in a structured, digital form, making the data processing step more painful. Still, the legally binding character of freedom of information requests often makes them an invaluable tool in the process of gaining access to financial data.

    +

    FoI requests may be necessary when you want to get more detail on the projects that government money is funding. Often the transactional spending data released will include only a brief description of the project, if at all. To get more information about it, you might need to submit an FoI request. For instance, if you have the high level payment information for a contract that includes the recipient, location and total amount, but you want to know the details of the contract deliverables, you will probably need to submit an FoI request for the full contract.

    +

    A good example of this process is the Sunlight Foundation's request for information on the Airport Improvement Program in the United States. The program accepts applications from airports around the country for infrastructure improvement grants, such as repaving a runway. Each project is assigned a safety priority rating and is prioritized in a queue. The high level spending information for this program was available in USASpending.gov, but since the priority ratings are specific to this program and not spending data in general, they were not included in that dataset. The Sunlight Foundation submitted a FoI request for the full dataset, including the priority ratings. After that, they were able to determine when airports with low priority projects were getting money, and how often. So the lesson is, if you see some interesting patterns in your high level spending data, don't be afraid to dig deeper and ask for more detailed program information.

    +

    Wanting to submit a request, but not sure where to start, who to address your request to or how to write it? Access Info (http://www.access-info.org/) are an organisation who work to help people obtain the information they require from the public bodies that hold it. They have also produced a toolkit (http://www.legalleaks.info/toolkit.html) to using FoIs. It's primarily aimed at Journalists, but most of the tips are equally relevant for CSOs.

    +

    Data scraping

    +

    Unlike open data or freedom of information requests, data scraping does not rely on the cooperation of government authorities for acquiring machine-readable documents. Scraping refers to the process of transforming unstructured documents - online database interfaces, PDF files or even printed documents into a more structured form (see section below on Optical Character Recognition for more tips on how to do this). Many tools and techniques enable such refinement by re-interpreting documents generated for humans into a more structured form which can be analysed, aggregated and distributed in new ways.

    +

    While there is an increasing number of easy-to-use scraping tools which do not require much technical knowledge, more complex processes - such as the automated scraping of thousands or millions of web sites or the mass interpretation of PDF files require some programming so that the process can be fully automated and produce reliable results.

    +

    In some cases, the only way to gain access to a set of figures is through the digitization of printed material. While scanners and optical character recognition (OCR) software can be used to import such documents, the high cost and low data quality generated through this approach often it an unattractive one.

    +

    A very accessible guide to scraping has been published (https://leanpub.com/scrapingforjournalists). Again, it brands itself at journalists (they're getting a little spoiled aren't they?), but scraping is scraping and so a worthwhile read for CSOs.

    +

    Another great source for tutorials, as well as being a tool itself to help with scraping is ScraperWiki. Using Scraperwiki (https://scraperwiki.com/) has the additional benefit of anything that you scrape being available to others, as most of the scraped data goes into a public data store.

    +

     

    +

    Getting data out of scanned documents

    +

    When you deal with scanned documents, the crucial step in the extraction process is to have the computer attempt to recognize any characters - letters, numbers and other signs. Optical character recognition (OCR) software is built to do this, accepting scanned pictures and PDF documents as an input.

    +

    There are both commercial software products for OCR (such as ABBYY FineReader, http://finereader.abbyy.com/), and some open-source software packages, such as Google's Tesseract (http://code.google.com/p/tesseract-ocr/). In general, the quality of all automatic recognition is limited, and you should make sure to cross check any numbers coming from scanned material against the printed documents.

    +

    Keeping the data around

    +

    As you retrieve data from the government (or other sources), it's easy to just consider the websites it has been released on as a permanent resource. Still, experience has shown that data does go away: whether it is through government re-designing its web sites, new policies that retract transparency rules or simple system failures.

    +

    At the same time, downloading complete copies of web sites - a process called mirroring - is a fairly well-established technique that can easily be deployed by civil society organisations. Mirroring involves an automated computer program (for a list see: http://en.wikipedia.org/wiki/Web_crawler) harvesting all the web pages from a specified web page, e.g. a ministry home page. In most cases, it is also possible to find old versions of web sites via the Internet Archive's Wayback machine (http://archive.org/web/web.php), a project that aims to create up-to-date copies of all public web sites and archive them forever.

    +

    Cleaning data

    +
    +

    <p >Data Processing Pipeline

    +

    As you acquire government spending information, you will notice that such data often has many inconsistencies: program names are used inconsistently, financial amounts will be stated in badly formatted numbers, while some data may not be usable at all due to file corruptions. In short: data always needs to be cleaned and processed. In fact, processing, augmenting and cleaning the data is very likely to be the most time- and labour-intensive aspect of your project.

    +

    Processing stages for data projects

    +

    While there are many different types of data, almost all processing can be expressed as a set of incremental stages. The most common stages include data acquisition, extraction, cleaning, transformation, integration, analysis and presentation. Of course, with many smaller projects, not each of these stages may be necessary.

    +

    In this process, each stage fulfils a unique purpose

    +
    • Acquisition describes gaining access to data, either through any of the methods mentioned above or by generating fresh data, e.g through a survey or observations.
      • +
    • In the extraction stage, data is converted from whatever input format has been acquired (e.g. XLS files, PDFs or even plain text documents) into a form that can be used for further processing and analysis. This often involves loading data into a database system, such as MySQL or PostgreSQL.
      • +
    • Cleaning and transforming the data often involves removing invalid records and translating all the columns to use a sane set of values. You may also combine two different datasets into a single table, remove duplicate entries or apply any number of other normalizations.
      • +
    • Analysis of data to answer particular questions we will not describe in detail in the following chapters of this book. We presume that you are already the experts in working with your data and using e.g. economic models to answer your questions. The aspects of analysis which we do hope to cover here are automated and large-scale analysis, showing tips and tricks for getting and using data, and having a machine do a lot of the work, for example: network analysis or natural language processing.
      • +
    • Presentation data only has impact when it is packaged in an appropriate way for the audiences it needs to aim at.
      • +

    As you model a data pipeline, it is important to take care that each step is well documented, granular and - if at all possible - automated. This is particularly true when processing more complex datasets, such as transactional government expenditure which has been scraped from a government website.

    +

    Data provenance

    +

    Good documentation on data provenance (the origin and history of a dataset) can be compared to the chain of custody which is maintained for criminal investigations: each previous owner of a dataset must be identified, and they are held accountable for the processing and cleaning operations they have performed on the data. For Excel spreadsheets this would include writing down the steps taken in transforming the data, while advanced data tools (such as Open Refine, formerly Google Refine), often provide methods of exporting machine-readable data containing processing history. Any programs that have been written to process the data should be available when users access your end result and shared as open-source code on a public code sharing site such as GitHub.

    +

    Tools for documenting your data work

    +

    Documenting the transformations you perform on your data can be as simple as a detailed prose explanation and a series of spreadsheets that represent key, intermediate steps. But there are also a few products out there that are specifically geared towards helping you do this. Socrata is one platform that helps you perform transforms on spreadsheet-like data and share them with others easily. You can also use the Data Hub (pictured below), an open source platform that allows for several versions of a spreadsheet to be collected together into one dataset, and also auto-generates an API to boot.

    +
    +

    Normalizing data

    +

    Data that comes from the government is often generated across multiple departments by hand. This can result in inconsistencies in what kinds of values or formats are used to describe the same meaning. Normalizing values to be consistent across a dataset is therefore a common activity.

    +

    First, you want to start by finding all of the distinct ranges of values for the different columns in your dataset. You can accomplish this by using a database query language (such as SQL's DISTINCT), or by simply using the 'filter' property on a spreadsheet program.

    +

    For example, if you have a spreadsheet with contracting data, and one column is 'Competed?', you would expect the values to be 'yes' or 'no'. But if this spreadsheet is an amalgam of spreadsheet data from multiple users and departments, your values could vary among the following: 'Y', 'YES', 'yes', 1, 'True', 'T', 't', 'N', 'NO', 'no', 0, 'False', 'F', 'f', etc. Limiting all of these potential values to two clear options will make it easier to analyse the data, and also easier for those who follow in your footsteps.

    +

    Especially with financial data, numbers can be formatted several different ways. For example, are your negative values represented with a '-' or placed inside '( )' or possibly even highlighted in red? Not all of these values will be easily read by a computer program (especially the color), so you'll want to pick something clear and consistent to convert all your negative values to (probably the negative sign).

    +

    Is all your numerical data measured out in ones or is abbreviated in thousands? Especially with budget data, order of magnitude errors are not uncommon when one department thinks they're reporting in thousands or millions by default but others expand their data all the way to the ones place. Are some values in scientific notation (e.g. 10e3938)? Make sure all your values are consistent, otherwise your analysis could contain serious errors.

    +
    A column of data requiring name normalization
    +

    Data quality

    +

    Data cleaning and data quality go hand in hand. You may find it helpful to conduct data quality tests after your analysis in order to measure your confidence in the data, or before to decide whether you want to do the analysis at all. Data quality can mean a few different things but here or some key factors to consider.

    +

    Completeness

    +

    Is all your data there? If you have a time series of some data, do you have approximately the same number of rows for each year or are there major outliers? An extremely high or low number of rows that looks out of place may indicate duplicate or lost rows. Or it could indicate a change in how data was reported at that time. Are there values for each column where values are required (according to the documentation)?

    +

    Accuracy

    +

    Does your dataset match up with other available data out there? This can be difficult to confirm, but it is often useful to employ a sampling methodology. Select a significant sample of your records (3-5%) and decide which values you can check for accuracy. If your data is transactional contract data, you could request the full contracts for your sample records and compare the structured data you have with the prose of the contract to confirm that it is correct. You can sometimes check contract and grant amounts from journalistic outlets and even company websites.

    +

    Timeliness

    +

    Sometimes governments can lag behind in their reporting. Even if a dataset says it's for a specific time period, do you notice a long tail of records closer to the end of the time frame? If so, you may want to restrict your analysis to the subset of data where you have a healthy sampling.

    +

    Case studies

    +

    Some projects have gone beyond simple assessments of data quality to ensure confidence in the underlying result of a project, but are projects completely about data quality themselves!

    +

    The Open Knowledge Foundation recently created a data quality monitor for UK transactional expenditure data (http://openspending.org/resources/gb-spending) which has been developed in cooperation with the data.gov.uk team. Each imported file is documented, highlighting any mistakes in the data structure or formal representation of the data. More information at:

    +

    The Sunlight Foundation has run a yearly report called Clearspending (http://sunlightfoundation.com/clearspending/) for the past three years that assesses the quality of the data reported in the US federal spending dashboard, USASpending.gov. The results haven't been very good but have highlighted the need for data standardization and oversight of the data release process.

    +

    As an even more thorough technique, little programs and tests can be written to check data validity along the processing pipeline by testing certain assumptions about each record in the dataset. If you do not have the option of creating such tests, you may still be able to create a spreadsheet in which you record known defects and issues with the data, including source files that did not open and inconsistent column values.

    +

    The most powerful data quality tool is much more simple, though: sharing the data. Without access to your source data, others will have to trust your word blindly. Therefore, sharing both the source and processed forms of your dataset is essential to discussing data quality.

    +

    Categorization and reference data

    +

    One of the most powerful ways of making data more meaningful for analysis is to combine it with reference data and code sheets. Unlike transaction data - such as statistical time series or budget figures - reference data does not describe observations about reality - it merely contains additional details on category schemes, government programmes, persons, companies or geographies mentioned in the data.

    +

    For example, in the German federal budget, each line item is identified through an eleven-digit code. This code includes three-digit identifiers for the functional and economic purpose of the allocation. By extending the budget data with the titles and descriptions of each economic and functional taxonomy entry, two additional dimensions become available that enable queries such as the overall pension commitments of the government, or the sum of all programmes with defence functions.

    +

    The main groups of reference data that are used with government finance include code sheets, geographic identifiers and identifiers for companies and other organizations:

    +

    Classification reference data

    +

    Reference data are dictionaries for the categorizations included in a financial datasets. They may include descriptions of government programmes, economic, functional or institutional classification schemes, charts of account and many other types of schemes used to classify and allocate expenditure.

    +

    Some such schemes are also standardized beyond individual countries, such as the UN's classification of functions of government (COFOG) and the OECD DAC Sector codes (http://www.oecd.org/dac/aidstatistics/dacandcrscodelists.htm). Still, the large majority of governments use their own code sheets to allocate and classify expenditure. In such cases, it is often advisable to request access to the code list versions used internally by government, including revisions over time that may have changed how certain programmes were classified.

    +

    A library of reference data that can be re-used across different projects and it is a valuable asset for any organization working with government finance. Sharing such data with others is crucial, as it will help to enable comparable outputs and open up options for future data integration. Existing repositories include the IATI Standard (http://iatistandard.org/) and datahub.io.

    +

    Geographic identifiers

    +

    Geographic identifiers are used to describe administrative boundaries or specific locations identified in a dataset. While some regional classifications (such as the EU NUTS) are released on the web, there is also an increasing number of open databases which contain geographic names - including geonames.org and the recently developed world.db.

    +

    Another related technique is the process of reverse geo-coding: translating a human-readable address into a pair of coordinates. Services like nominatim (http://nominatim.openstreetmap.org/) will not only enable users to generate precise maps of projects in a region, they will also return the responsible administrative boundary for many coordinates. This means that projects which are given by precise address can also be aggregated by state, region or any other geographic unit.

    +

    Additionally, many countries have shapefiles of their political and geographic districts available (usually through the census or interior bureaus) that can be imported into custom mapping applications, like TileMill (http://mapbox.com/tilemill/).

    +

    Company and organisational identifiers

    +

    As you look into spending data that includes recipients outside the government, you'll find companies which act as suppliers to government, but also other types of entities including charities, associations, foreign governments, political parties and even individuals which act as recipients of direct assistance.

    +

    Identifying such entities is notoriously hard, since the only information kept by government is often a simple name (which may not uniquely identify the beneficiary, for example "MS"). While most (if not all) countries maintain company registers which assign some type of unique identifier to a company, these databases are often not accessible in bulk and not used coherently across different parts of government. Alternative identifiers - such as tax identifiers and company IDs from private business information suppliers (such as Dun & Bradstreet in the US) - further complicate this process.

    +

    As an alternative, open registries are beginning to compile organisational identifiers in a form that is easy to re-use and thus enables the sharing of databases which have been augmented with such information. OpenCorporates.com (http://opencorporates.com) is a startup that collects information from companies world-wide and provides a convenient API to match datasets with the list of known countries. The IATI project for aid transparency is working towards similar standards for other organisations, such as foreign governments and charities active in the development space.

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch008_anaysis.html b/examples/openspending/content/resources/handbook/ch008_anaysis.html new file mode 100644 index 00000000..73296757 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch008_anaysis.html @@ -0,0 +1,75 @@ +--- +title: Analysis +layout: handbook +--- + +
    +
    + +
    +
    +

    Analysis

    +

    +

    Data Processing Pipeline

    +

     

    +
    Dear reader - we know you are the experts. We're not going to teach you all the basics, but this section's aim is to show you how technology can help speed up, scale up or simplify your working practices. For this reason in this section we will say only a few words about statistical methods, even though many of you will be very familiar with those.
    +

    Once you have your data in a suitable format to be dealt with, much of your analysis can be automated with a custom program or inside spreadsheet software. Obviously the deeper your data dive is, the less automated things become, but having a standard set of post-processing, analytical steps to run on a dataset helps make sure you haven't missed anything. Some of the steps may seem basic, but it always helps to have a checklist!

    +

    Sorting data

    +

    Possibly the simplest way to find some interesting points is to sort by a column or category. In spending datasets, it's common to rank data by the largest transactions and look at the beneficiaries. This also gets you the maximum or minimum values of a column, which could be interesting.

    +

    Basic statistical analysis

    +

    Statistical analysis such as calculating percentiles or the average of a dataset, are essential to characterize your data. These are available in most programming languages as well as most spreadsheet programs like excel. A more complete list of statistical tools and resources can be found at the resources chapter.

    +
    • An average is a measure of central tendency in a sample. It can be described as the "typical" or representative value of a data set. Knowing the average helps detect whether any single payment is atypical.
      • +
    • A percentile is the value of a variable under which a certain percentage of the data fall. For example, if a transaction amount is in the 90th percentile, then it is larger than 89% of the transactions in the dataset.
      • +
    • A standard deviation measures how far away a value is from the average. If your standard deviation is low, then it means your row values are clustered together. If it's high, then they are spread out. By checking for values that are one, two, or three standard deviations outside the mean, you can detect outliers. Outliers could be errors or extremely interesting data points.
      • +

    Summarizing or aggregating by categories

    +

    Sometimes data is available at a very high level of detail, making it hard to see the big picture. It's a good idea to have a standard set of categories to aggregate by, such as geographic unit, functional classification, account number, fiscal year or other temporal unit, government department, or even recipient.

    +

    Normalization for comparability

    +

    In order to be able to compare data across time or other parameters, you may need to systematically adjust the data values. Here are some examples.

    +
    • Deflation or inflation. Expenditure data is naturally obtained with current (or nominal) money value, but such data is not directly comparable across different years since inflation affects the value of money. In order to compare currency across years, it is necessary to deflate or inflate the value according to a base year. For example if we wish to compare the budget of a program in dollars from 2000 with the same program in 2012 we must adjust the value of the 2000 data and the 2012 dollars to a base year. Deflator and inflator factors are calculated by central banks in most countries and a comprehensive list by country can be found at the World Bank website (http://worldbank.org)
      • +
    • Scale and normalize values. When we wish to compare patterns of budget programs sometimes the difference in scale between budgets may prove difficult to display in a graphic form. In such cases is useful to compare percentages instead of absolute numbers. For instance, it is very common to express budget and spending data as percentages of a country's GDP, instead of in absolute numbers. If you're mapping spending, make sure to scale your data by population density. Otherwise, your map will most likely just be a map of where your populace lives.
      • +
    You may want to visualize your data to get better understanding of your initial assumptions. This may save you a lot of time in case your initial assumptions were not good, and may also help you to get more insight into new findings your data might contain but previously were not clearly visible. For this purpose you can use scatter plots, line graphs, heat maps, bar charts.
    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch009_ngo-ngo-collaboration.html b/examples/openspending/content/resources/handbook/ch009_ngo-ngo-collaboration.html new file mode 100644 index 00000000..10cda115 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch009_ngo-ngo-collaboration.html @@ -0,0 +1,91 @@ +--- +title: Using technology in your work +layout: handbook +--- + +
    +
    + +
    +
    +

    Using technology in your work

    +

    As you move into more advanced data projects, you may find yourself in a situation where a research project turns into an effort that involves coders, designing databases and web sites. It is important that you take a step back and realize that you are now running not just an advocacy effort, but also an IT project.

    +

    There are many difficulties which CSOs face when developing software. Some common issues include:

    +
    • Difficulties in finding qualified developers that want to contribute to your projects at a reasonable rate, as well as in the communication between CSO staff and developers.
      • +
    • Clearly communicating the requirements for the software so that both non-technical and technical staff share a vision for the outcome of the project.
      • +
    • The estimation of time and resources for particular tasks, especially how to handle projects that drastically overrun the timeframe and funding they were initially assigned.
      • +
    • Evaluating the work of developers to ensure that the product that has been delivered is according to what has been agreed, especially in small projects with only a single coder or when working with external contractors.
      • +
    • Maintaining the project after the main development period has finished
      • +

    Starting from scratch vs. re-using Components

    +

    It is always going to be more costly and riskier to develop something from scratch than to customise something that already exists.

    +

    You should make the software behind your projects open source. If many other organizations do the same, this allows code to be reused across jurisdictions. Not only does this ease the financial burden, but it helps create the expectation in populaces around the globe for the high quality engagement tools that their neighbouring country has access to.

    +

    That's not to say you should never develop something new - just ask around first, and make sure that what you are asking is technically feasible.

    +

    Commissioning New Software

    +

    Purchasing software is more closely related to having a piece of clothing made than to buying chairs. You can give the designer a basic vision of what you would like, but you will always need to come back to make sure it really fits, and your thoughts may change when you see things in practice. If you have an arrangement with your tailor which allows you to first specify the general idea, and a couple of other appointments for fittings and trials, you'll probably end up with a better and more creative result than if you tried to design the whole thing and it was simply unveiled to you at the end. You'll also feel more in control and it may even be quicker to do design and implementation in parallel.

    +

    Defining requirements & Designing the solution

    +

    Define the basic components of your project and prioritise them by their importance. As the developers start working on one of these chunks, you can then break it up into more specific tasks based on your evolved understanding of the project. A popular technique for this purpose are user stories, small narrative pieces that describe each problem: "As a [web site visitor] I want to [be able to see a supplier's contracts] so that I can [understand what services they provided to government]". The key to these stories is that they describe the actual user need, not the details of the solutions that you have envisaged. While you should of course discuss those with the developers as well, defining solutions is mainly the job of the developers, not the project manager.

    +

    Implementation: iterations and milestones

    +

    A saying amongst developers goes: "Walking on water and developing software from a specification are easy if both are frozen." As your software project is progressing you will likely realize that the specifications you have given need to be revised or extended. Yet by modifying the requirements you are essentially shifting the ground on which developers are executing - meaning they will have to stop their work to adjust. To prevent such changes from freezing all development, the process of introducing changes and additional requirements needs to be structured.

    +

    Iterations are periods of a defined length - often two or four weeks - during which developers are tasked to execute a set of previously selected user stories or requirements. Before the iteration starts, developers have to pull in the work from a list of tasks (a so-called backlog) prepared by the project manager, committing themselves to delivering those tasks within the agreed period. Crucially, project managers are not allowed to extend or revise the scope of an iteration while it is ongoing (unless they want to declare it failed). This method ensures that changes are introduced in bulk and understood by the team. This approach mandates the opposite of the more common unstructured communication between managers and developers, e.g. emails with unsorted lists of change requests which tend to be ignored and lead to confusion.

    +

    Whenever you consider an additional requirement, be sure to consider if it is realistic within the resources you have available. "Scope creep", the progressive extension of a project during its development, is a common cause of project failure. By becoming more and more ambitious, the project finally ends up with no usable product at all. To avoid scope creep, make sure to have a storage area for long-term ideas. Also make sure that developers accept additional tasks through a pull process, and not by having them pushed into their workflow.

    +

    Maintenance

    +

    Make sure to budget a for ongoing maintenance after the end of your project. Who is going to guarantee that the servers stay online? Who is going to fix a typo? It is unlikely that your project will remain entirely static after its initial development, so you should have an explicit agreement with the developers regarding future support. It is also useful to collect feedback after the projects release to commission a small number of additional days when enough additional work has accumulated.

    +

    Roles and how to find developers

    +

    The key ingredient to a successful software development project is having the right people on staff or as contractors. Depending on the scope and type of your project, you may need a variety of skills - these are some of the common descriptions:

    +
    • Web designers typically produce designs and layouts for web pages, often initially in a graphics program like Adobe Photoshop. Most, but not all, web designers then translate their designs into web markup (HTML, CSS).
      • +
    • Web developers are more technical. They produce interactive web interfaces such as search masks, browsers or specific form-based operations. They often use programming languages such as PHP, Ruby on Rails, Python or JavaScript.
      • +
    • Visualization designers develop graphics that represent quantitative information. A key distinction here is between non-interactive graphics (i.e. static images) and interactive visualizations, which often require some programming. There are still very few designers who design interactive visualizations, so rates may be relatively high.
      • +
    • Software developers are even more technical, developing backend software for data processing or acquisition. They are experienced in the use of database software (such as SQL databases) and programming languages such as Ruby, Python or Java.
      • +
    • Data scientists and statisticians produce analysis based on large sets of data, detecting tendencies and outliers in the dataset. They are not usually expected to produce front-end applications, but may produce software in the process of analysing data.
      • +
    • Usability experts and user experience (UX) designers think about the way your user will interact with your site answering questions such as 'is it obvious from the landing page what the purpose of this site is?'
      • +
    • Testers try and break things to test their robustness. This is particularly useful e.g. if you think your project will receive a lot of traffic as a result of a media campaign, you want to know your site can survive the hit
      • +

    Good places to look for developers

    +

    The easiest way to meet developers is through community meetups, such as hackdays. During such events, coders meet up to cooperatively develop prototypes of new software. To meet volunteer developers who can help you make sense and unleash the power of government spending and budgets, it's wortwhile to investigate events such as Random Hacks of Kindness (http://www.rhok.org), Data Kind (http://datakind.org/) and TechCamps (http://techcampglobal.org/).

    +

    There are a few ways you can discover if there is a hackday in your area. One way is to search on Lanyrd (http://lanyrd.com/search/?q=hackday&context=future) or set up an account on that system and request that you are alerted when there is a hackday in your area. Another approach is join mailing lists of organisations that might help you find developers e.g. the Open Knowledge Foundation lists (http://lists.okfn.org/mailman/listinfo) or the Sunlight Labs mailing list (https://groups.google.com/forum/#!forum/sunlightlabs).

    +

    What do things cost?

    +

    It's impossible to give concrete guidelines on how much a project should cost. Developers' salaries are generally quite high for a country's average, but vary very strongly from country to country. Worried about your project spiraling out of control? We'd recommend agreeing on a price per iteration, and it may be a good idea to draw up a contract which allows you to break it off if you are not happy with the work at the end of an iteration. Plus, you can generally also find a friendly developer to glance over a quote from a company for a sanity check.

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch011_defining-the-scopetopic.html b/examples/openspending/content/resources/handbook/ch011_defining-the-scopetopic.html new file mode 100644 index 00000000..a6d634a9 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch011_defining-the-scopetopic.html @@ -0,0 +1,98 @@ +--- +title: Presentation and engagement +layout: handbook +--- + +
    +
    + +
    +
    +

    Presentation and engagement

    +

    While raw financial data is valuable just as a tool for research, analysis and investigation, it can also be directly presented to an audience in a wide variety of ways. Such presentations can take the form of press releases in which data is used to make an argument, interactive infographics which condense information to visually represent a key finding or even searchable on-line databases that allow each user to find spending in their neighbourhood.

    +

    +

    Clarifying your topic

    +

    "It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away" - Antoine de Saint Exupéry

    +

    If you are reading this, you probably already have a fairly clear idea of the topic that you want to investigate and present. Still, it is useful to look at different ways of defining your subject. In some cases, you may merely want to give an overview of a particular area, e.g. by showing the proportions of government spending which go into the various functions of government. In other cases you may have a particular hypothesis that you want to portray or a particular journalistic story that you mean to tell, such as a case of systemic corruption that becomes evident only as you contrast one department's spending patterns with that of another one.

    +

    Sometimes there is a temptation to get lost in complexity. Using all the data and tools available to you to their full potential can force an almost endless amount of information onto our audiences. The benefit of doing so is limited, however: people will not understand the point that you are making, or, even worse, ignore you.

    +

    At the same time, technology can also be used to do the opposite: focus in on the key points, hiding other detail. Selecting where to go into depth doesn't have to be the author's responsibility any longer, it can be a choice that the user of data-driven applications is empowered to make based on their own interests. The prerequisite for using technology to create a clear focus is answering the question: What is the core of what we mean to present, what is the essence of the narrative our project is intended to convey?

    +

    Setting an objective

    +

    As you focus the topical core of your project, you also need to think about the objectives of your project: is it intended to inform the general public about an aspect of government budgeting or spending practices? Are you making a specific case to government, e.g. by proposing a new policy which you have gathered evidence and developed arguments for?

    +

    Your project can also serve as a forum for discussions, which would usually require you to facilitate the debate to maintain ongoing engagement.

    +

    The objective may be to establish a network-based, collaborative effort, e.g. to build a community that may review documents released by the government or evaluate the conditions of public contracts. Such modes of engagement require extensive design to encourage users to contribute in a way which best matches their particular skills and interests.

    +

    Some example of working with the public to collaboratively clean up financial data is Linked Gov (http://linkedgov.org/). This is a platform that will allow the public and civil servants to access and do small tasks that will clean government financial data.

    +

    Targeting an audience

    +

    The efficiency of projects based on spending data depends heavily on how well you have defined and understood your audience. If your message is too general for a specialized audience, it will be ignored. At the same time, a complicated and very technical presentation will usually fail to address broader parts of the population. Whether it be that your main audience is a specific constituency, government officials, the general public or a combination of those, it is important that the media and methods you apply help to get the message(s) across to the audience that you are targeting.

    +

    The general public

    +

    Is the purpose of your campaign to stimulate discussion amongst the general public? Unless your organization has exceedingly well-developed outreach channels, you will not usually be able to address the general public directly. Still, thanks to the social media and internet-based direct communication, many organizations can now contact and involve citizens who have shown some interest in their work directly. In cases where your main audience is the general public, simplicity is a must. It is necessary that the vast majority of people get a grasp of what you are trying to communicate. In these cases you may need to emphasize the visual presentation of the data that gets quickly and succinctly to the point, such as infographics. See the case study at the end of this chapter for how an organisation in Nigeria used Social Media to reach out to the general public.

    +

    Mass media

    +

    Unlike direct channels, broadcast and print media require you to first convince an intermediary of the relevance and quality of your message. Media are often the best channels to use when we want to create social pressure on governments to change some budget issue.

    +

    Community and constituency

    +

    This may include people and organizations with which you have previously cooperated or which are active in the same domain. Unlike the wider public, they are more likely to be interested not just in your conclusions, but also in the data and tools that you have used to reach your conclusions. Creating an open space for your community to discuss your work is essential in getting relevant feedback. When you need to reach to your constituency it may be necessary to provide a broader context, such as a full report including methodology, conclusions and possible paths of further discussion.

    +

    Government

    +

    Officials are often a target of your outreach, since they are in the position of actually deciding and implementing changes in policy. While they can be reached through public pressure and the media, your organization or some of its constituency may also choose to contact governments directly. In the case of trying to reach government officials, a balance of the previous approaches is very important. Meetings with government officials are rare and short-lived. Condense your main argument and supporting evidence into one page or less, and include a clear action that you want them to take.

    +

    Obviously, on any given issue you may choose to involve any set of these groups, but it is still important to have a clear picture of who they are and what their incentives motivate them.

    +

    +

    Case study

    +

    BudgIT Nigeria on social media

    +

    Data is invaluable. To the uninformed person where information asymmetry thrives, access to data illuminates the path to facts and provokes emotions that trigger results. Nevertheless, poor handling of data puts valuable facts in an opaque structure that communicates nothing. Data could be in a maze of thick document riddled with complex terms or iterations of figures that doesn’t connect the user. Not being able to promote discussion nor give provide contextual understanding, data may could be worthless.

    +
    +

    Nigeria returned to democracy in 1999 after lengthy years of military rule. Data under the barrel of the gun was an exclusive preserve of rulers as probing the facts behind data is taken as an affront to authority and strive to question the stained reputation of the junta. Civil servants were bound by the Official Secrets Act not to share government information hereby putting the citizens in the dark. Even after thirteen years of return to democracy, there is clearly a gap in accessing public data with some government officials still stricken with military era hangover. Data especially in terms of public expenditure communicates little to the larger sections of public who are not versed in financial accounting nor have requisite understanding to evaluate the complex arithmetic.

    +

    BudgIT, creative start-up sees a huge opportunity in using creative data visualization to stimulate interests concerning public expenditure. Understanding the ubiquity of the mobile device within the Nigerian locality and the increasing number of Nigerians online, BudgIT sees the opportunity to engage Nigerians and explain public expenditure in a simpler way. This thrives on building engagement across all platforms, encouraging the community to action via NGOs and reaching out to everyone to promote citizen participation. This is about making public data a social object and building an extensive network that demands change.

    +

    Building The Community

    +

    Across our interactions with users, we see a gulf in understanding what the budget is and what the citizens usually expect. We have engaged over 10,000 Nigerians over the budget and we profile them into three to ensure optimum value is delivered. The categories are briefly explained below:

    +

    Singular Users: These are users who want information in simple and quick format. There are interested in data but not in a deeper scale of analyzing the content or probing further. A series of simple tweets is enough for them to retell the stories or interactive applications that give a snapshot.

    +

    Closed Loop Users: A set of users who stimulate a discussion pattern, engage the data channel effectively to increase their knowledge of subject matter or challenge the assumptions of data. These set of users are adherents of the platform via social media, hence they ensure in building the platform through feedback systems or referral to their social connections.

    +

    The Data Hog: This set of users want the raw datasets to rework visualization or do extensive analysis for personal or enterprise purposes. We simply give them the data for their definitive purposes.

    +

    Engaging the Citizen

    +

    Across every society lies a literacy span and engaging every component is highly critical to the societal growth and stability. There is always a storyboard in every life that volumes of data can be matched with. People constantly want to be more informed especially concerning issues that they find difficult to understand. Engaging citizens is to take critical analysis of the target users at a time and itemize the possible profiles. A look at user profile demands a thorough analysis of their empathy, attention and insight towards the data available to them. What does the Nigerian citizen care about? Where is the information gap? How quickly can we reach out to them and place data in the storyboard of their lives? A critical understanding of the user’s psychology and the perceived response to the data is the first needed level of analysis. BudgIT’s immediate reach is to the average literate Nigerian connected to online forums and social media. Most online users amidst the array of interests in gaming, reading and sharing social connections within a limited timeframe will definitely need data in a brief and concise manner. After a snapshot of data either as a tweet or infographics, there’s an opportunity to build linked data on other platforms where the big picture can be set and interaction can be enhanced. <p >An important angle of visualization to us is understanding the data appreciation level of the users. Complex diagrams, superb infographics and aesthetic interactive applications might not convey the exact meaning to user based on his/her previous approach to data. Data vizualization needs to take into consideration how users can easily grasp the vizualized data and subject it to personal interpretation. A good visualization transfers knowledge and mostly important brings forth a story the user can easily connect with.

    +

    For us in BudgIT, our engagement model is anchored on the following:

    +
    • Stimulating discussion around Trends: In engaging with users on public data, BudgIT keeps track of online and offline discussions and seeks to provide data to enliven the interactions. A glaring example was the fuel strikes in January 2012 where there was a constant agitation among the protesters on the need to reduce the size of governance. BudgIT tracking the discussion via social media in 36 active hours quickly built an app that allows citizens to cut the Nigerian budget. The huge response with over 3,000 users who interacted with the budget using the app refined our engagement model. We keep looking for trends in the polity and matching it with relevant data quickly rendered into tweets or infographic display that quickly  extends our influence.
      • +
    • Constructive feedback mechanism and Balanced outlook: Data speaks volumes and individuals subject it to personal interpretations. In the engagement with users, feedback is enabled through discussion boards or retweets. Most users throw up discussions that tend to ask about stories behind the data and seeking opinions of BudgIT. Its of utmost priority to ensure that opinions only explain the facts behind the data and does not conform to individual disposition to the subject matter. It’s most necessary to build up feedback channels and engage the users creatively to ensure the community built around the data is sustained.
      • +
    • Localize Outlook of Data: For a dataset targeted at a particular group, BudgIT is building competency to localize its content and promote a channel of discussion that connects to the users. This involves taking a cultural outlook at the icons, symbols, objects and language to ensure that engagement concerning the budget flows seamlessly. This phase of engagement is with the grassroots who mostly don’t have access to Internet but only possess SMS-based mobile phones.
      • +

    After making the public expenditure data available in an easy-to-read format as shown on our portal (http://yourbudgit.com), we reach out to the citizens through the civil society organizations to ensure that citizens monitor capital projects in the budget. We also plan to develop a participatory framework where citizens and government institutions can meet in town halls to define key items in the budget that needs to be prioritized. Once we get citizens to be aware of capital projects in the budget and connect them with civil societies where BudgIT is not located, citizens can track report projects and report status. Ensuring that citizens of any literacy span are armed with data and possess a clear path to demand action, BudgIT is crossing the rubicon from open data to open action.

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch012_selecting-methods-and-tools.html b/examples/openspending/content/resources/handbook/ch012_selecting-methods-and-tools.html new file mode 100644 index 00000000..01961983 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch012_selecting-methods-and-tools.html @@ -0,0 +1,116 @@ +--- +title: Selecting methods and tools +layout: handbook +--- + +
    +
    + +
    +
    +

    Selecting methods and tools

    +

    The key to using data to tell people about your issue is to choose a really good story. The key to finding a good story is to formulate the right question for your audience.

    +

    Be guided by what interests you, as this stands a good chance of interesting other people. If you don't feel confident about your judgement then try lots of things that you consider interesting and see if it makes impact on the people you care about. Are they sharing your work on social media? Do they email you or link to your work from their own sites? If the answer is yes then you are really making good data stories.

    +

    Of course rigour and accountability are essential and you should always include links to your methodology and data, but it doesn't need to obscure your headline. Telling a story is the difference between people not reading your work and making it all worth the effort.

    +

    If you've already got data products that you want to share, try disseminating them in a couple of different ways. In addition to the methods outlined below, try preparing primers or manuals for other groups that want to start working with datasets that you may be particularly experienced with. You can also organize workshops, meetups, and webinars to familiarize groups with the data you work with. If grassroots issue groups can internalize your data products and present them in a contextually relevant way to their user base, then both groups have gained something from the partnership.

    +

    This is a collection of (hopefully) inspiring examples where groups have connected to the public directly through their websites, social media, or through the mainstream media. If you are interested in making media but don't have the capacity then fear not! Check out the first section of the book on reaching out to other organizations for help or running your own project.

    +

    Communicating via the web

    +

    If you are creating or adding to a website that you want citizens to access directly then here are some of the ways you can really use budgets or spending data to increase the value of your organisation.

    +

    Firstly we highly recommend that if you have some really great data, that's a story in itself, then don't waste any time - put the data up on your website and write a great story to go with it. Make sure the full dataset is available to download, ideally in a couple of formats like csv and json, and explain your methods. Then encourage your supporters to use and play with the data, add comments and talk in your community site or social networks.

    +

    If you feel your data needs that little bit more work to make it interesting or digestible, here are some examples of portals, interactive infographics, images, reports, and a searchable dataset.

    +

    Infographics and images

    +

    The art of the infographic is to make the complex look simple. This can be invaluable if you can communicate huge amounts of data in one picture. It is always recommended, both as a boost to your credibility and to allow others to extend your work, to publish the complete data needed to recreate the image. This allows your community of supporters to create their own visualisation and check your work easily before republishing it.

    +

    Below is an infographic summarizing the history of the US Federal Budget, by the Congressional Budget Office (CBO) /http://www.cbo.gov/publication/42636). It shows the budget over time and as a percentage of GDP, and in the context of total spending and revenue. This was used to emphasise that the federal government's budget deficit for 2011 was the third-largest in the past 40 years:

    +

    +

    Images of spending

    +

    This "binder full of burgers" infographic was created to show the sandwich of spending for Barack Obama's 2012 presidential election campaign. The data is not especially novel but the graphic is fun and engaging.

    +

    burger full binders

    +

    Interactive infographics

    +

    An interactive infographic can add even more data to a single screen full of images. A simple example is a bar chart where hovering over a bar with the cursor results in a dialogue box that tells more detail, like the exact value of that bar.

    +

    It seems there are two types of interactive graphics that work very well. One is where the user finds their own path through a complicated landscape of data. For example a map of the world where you can zoom in to details of each country or regions within a country and discover some facts about that area. The other is where the infographic guides the user through a story. For spending data in particular, people often like the ability to "drill down" in an infographic, or in other words, start with a high level view of the data and then gradually view more detail about different sections that they can select.

    +

    In recent years we have seen the quality of interactive infographics grow and become easier to produce. Anyone with a Google account can now make maps (http://www.google.com/drive/start/apps.html#fusiontables) and charts (https://developers.google.com/chart/). Equally, the tools to manipulate and display data in more sophisticated ways have become more accessible with efficient and usable javascript libraries. Using these libraries requires a bit of technical skill, so you may want to reach out to other organizations, as described in the first section of this book.

    +

    Budget interactives

    +

    An interactive graphic gives you the perfect opportunity to allow people to choose their own budget. This interactive chart called Citizen Budget is from the non-profit Open North. It shows Canadian budgets and gives citizens the chance to play at being the official controller. They can cut spending in any area they choose but the budget has to balance.

    +

    http://citizenbudget.com/

    +

    Spending interactives

    +

    The World Bank have a spending database called Boost. It's a technical collaboration between the World Bank and various governments to digitize their spending data. It is especially targeted at countries that wouldn't otherwise have the technical resources. One of these countries is Kenya and the data has been displayed to great effect here:

    +

    +
    The examples given here for interactive infographics showing both budget and spending data are examples of where the user can navigate for themselves. Please let us know if you have examples of step-by-step guides to Budgets and Spending as it would be great to include them here.
    +

    Online reports

    +

    A report allows you to expand on points and show lots of different views of a dataset that would be just too cluttered on one infographic or in an interactive. A report is a good place for a step-by-step guide through a more extensive study.

    +

    An example of a very short report is from Enough is Enough (EiE), a coalition of Nigerians aiming to enhance citizen engagement and good governance. Their EiE report on the state of the Nation shows a very concise summary of the major spending corruption stories they identified that year, along with some links to sources.

    +

    Podcasts and videos

    +

    A well made podcast will almost certainly be shared widely given the right push. Describing the significance of your data and getting expert comments is so much easier to digest in a podcast than a long report - even if they are both very well written. Again, the technology to make radio has radically reduced in price in recent years.

    +

    The NPR Planet Money podcast rose to fame for its incredibly accessible, award-winning coverage of the 2008 Global Financial Crisis. Planet money now covers financial issues all over the world on its weekly podcast.

    +

    Online portal

    +

    A portal is suitable for a large and relatively frequently updated dataset. People in your community can then build web applications to pull the data. An example of a portal that might be used to add more context to spending data is the police.uk website. This is a complete view, updated monthly, of crimes committed in the UK, with details down to street level. The site also provides an API[15] which you can use to add new data to an app showing, for example, spending on police with crime levels overlaid on map.

    +

    Mobile technologies

    +

    There are a few free and open tools that will allow you to display your data through a mobile phone or tablet app. AKVO is a non-profit foundation and they describe (http://www.akvo.org/blog/?p=4822) how their AKVO FLOW (http://www.akvo.org/blog/?p=4836) platform works using free open source software to communicate your data through an Android mobile app.

    +

    Social media

    +

    Building a community around your cause in general and data analysis in particular is really valuable. Setting up a mailing list and creating a space to collaborate like on a wiki can lead to some really interesting social connections at relatively low cost can allow you to work with some really devoted and knowledgeable people. Some nice examples are the OKFN mailing lists (http://lists.okfn.org/mailman/listinfo) using Mail Man - http://www.gnu.org/software/mailman/index.html), wikis (for example http://www.wikispaces.com/).

    +

    There are also the big social media companies like Twitter and Facebook of course! These are great for reaching large numbers of people who have anything from a passing interest to complete devotion to your cause. When you join create a Twitter account or Facebook page with the name of your organisation. The strategy when joining these is to contact some high profile tweeters or facebookers to take on your cause or message. Make sure there are plenty of links back to your website when you Tweet or post to Facebook or other platforms. If you have a data release then create a package of the data, the story and any other resources (image downloads, reports etc) on a webpage and then tweet a link out with a link to that page. The same principle applies with Facebook.

    +

    Communicating with the press

    +

    When writing a press release keep it simple and very relevant to a story the journalist might like to run - leaving descriptions of your organisation or how you got the data to the very end or ideally include only a link to a relevant webpage. To be relevant you need to hook into something in the news cycle (http://en.wikipedia.org/wiki/24-hour_news_cycle) or offer up such a strong story it would create news. If it is the latter, then you might want to give it to a number of newspapers with an embargo so they will release the story all at the same time to create a really big impact.

    +

    If you are looking for news hooks then at the simplest level the data can relate to the time of year, maybe when there is a new financial phase or an election and the news is focused very much on spending and budgets. For example if it's a complete cleaned up version of all public spending, or budget or all the pension funds, just showing this data qualifies as something a journalist would find interesting - they may even have an in-house team working on the same thing so will be pleased to have the data quicker. Here the press release can be quite simply describing the data if the data is detailed and relevant enough to stand alone, along with a link to your portal and some instructions for using it, or a download of the data.

    +

    If you're wondering about how much extra work you need to do to present your data before sending out a press release then the same principles apply as when adding content to your website. If you have a great story from a great piece of analysis or just great data alone, then send the data and the story in a concise form with links through to the methodology on your website.

    +

    One final tip before describing what you might include in your press release: when working with journalists be sure to get a link to your website included in the piece.

    +

    Including images

    +

    If you do have a dataset with a story that would really benefit from an infographic then add it to the press release. Or if you have a mobile or tablet app, a podcast or report then add a screenshot of the most enticing view of it too. Ideally attach a small copy of the image to the email and give a link to a higher resolution version on your website.

    +

    It's likely the newspaper's graphics team will create a version in the newspaper's style so you should include the data for your graphic in a usable format (no PDFs).

    +

    The journalist may also choose to use your graphic as it is, so be sure to add a small discrete copy of your logo in the corner of the image or other citation embedded in the image.

    +

    Including an interactive infographic

    +

    Equally you may have made a descriptive interactive, in which case you might want to make it 'embeddable' on other websites. You may also ask for the news site to link back your own website, or include a link in the embeddable portion of the infographic.

    +

    Communicating with governments

    +

    In modern democracies we would advise CSO's to influence government decision making with evidence-based, factual, easy to understand policy proposals. These proposals can be related to almost any particular subject, from energy efficiency, labour policy, immigration, minority issues to public finance. Government relation to these policy proposals can be decisive in times of elections, and for this reason they cannot ignore them. Government capacity is limited, so you might find it takes a while to get a response from them.

    +

    For CSOs it is important to differentiate between Government's general goals, and policy specific goals. For example, every responsible Government has a general goal of maintaining a balanced fiscal policy. As we get more specific about this goal, stakeholders' opinions begin to differ. Governments are rarely very clear about their policy goals, as this would make it easier for CSOs to keep the Government accountable. Real policy goals can also be very different from what Government declares as its policy goals. For example, every Government is against tax evasion, but in many cases Governments indirectly support them, one way or another.

    +

    Common themes for communicating

    +

    To summarize, the key themes are: have a great story, be clear, be connected to lots of other people and be accountable by publishing methodology and data. These apply equally if you are communicating with the public or the media.

    +

    Being clear is making easily repeated points with the data, or describing a juicy data set in one sentence. If you can make the presentation fun, fascinating, or just beautifully simple then you are on to a good thing.

    +

    Being connected is making things technologically very shareable and then facilitating the spread of your message through your own community of supporters and social media.

    +

    Being accountable is being completely transparent about how you obtained, cleaned and analysed the data. This method has to be explained very clearly too, and the standards maintained for every release until your CSO has a good reputation of trustworthiness. But please don't let concerns about trustworthiness stop your CSO from exploring and having fun with the data in the first place, which is really something you need to consider when settling on a data story.

    +

    The final points are wherever possible, have fun telling your story and don't be afraid to experiment with different styles and tones but don't delay if you have a great dataset on your hands - let the world know!

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch014_resources.html b/examples/openspending/content/resources/handbook/ch014_resources.html new file mode 100644 index 00000000..a2be9c73 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch014_resources.html @@ -0,0 +1,84 @@ +--- +title: Resources +layout: handbook +--- + +
    +
    + +
    +
    +

    Resources

    +

    There are a variety of resources that you can use to accomplish some of the technical tasks described in the book. Some are geared towards users who are already experienced in a programming language, and some can be used by those with no experience. Most are helpful with analyzing and visualizing data.

    +

    For analysis and research

    +

    For visualisation and cleaning 

    +

    Free and Open Source:

    +

    Others

    +
    +
    +
    diff --git a/examples/openspending/content/resources/handbook/ch015_glossary.html b/examples/openspending/content/resources/handbook/ch015_glossary.html new file mode 100644 index 00000000..d1bafe03 --- /dev/null +++ b/examples/openspending/content/resources/handbook/ch015_glossary.html @@ -0,0 +1,103 @@ +--- +title: Glossary +layout: handbook +--- + +
    +
    + +
    +
    +

    Glossary

    +

    aggregated

    +

    data that is summarized using statistical methods or by simply summing its component parts

    +

    API

    +

    Application Programming Interface: a specification allowing two pieces of software to interface with each other, without either having knowledge of the inner workings of the other

    +

    backlog

    +

    a list of tasks prepared by the project manager

    +

    bulk data

    +

    refers to the entirety of a dataset, instead of the incremental amounts that may be retrieval via other means, such as an API

    +

    CSO

    +

    Civil Service Organizations: a legally constituted organization created by natural or legal persons that operates independently from any form of government in the interest of civil society. Also commonly referred to as NGOs.

    +

    CSV

    +

    Comma Separated Values: A format for transactional data that separates each column value by a comma. Can be imported and exported from most spreadsheet programs

    +

    database query language

    +

    a syntax for interacting with data in a database. Usually specific to the software being used

    +

    hacker

    +

    one who programs enthusiastically (even obsessively) or who enjoys programming rather than just theorizing about programming (like a 'coder' - but more energetic :) )

    +

    hackday

    +

    a session where hackers come together to answer a particular task

    +

    iteration

    +

    a period of a defined length during which changes to a project can be made. Usually ends in a review meeting to plan the next iteration.

    +

    JSON 

    +

    Javascript Object Notation: a data format that is relatively easy for humans to read and write and very easy for machines to read and parse

    +

    Mirroring

    +

    creating an entire copy of a website on another server, primarily for backup

    +

    MySQL

    +

    (pronounced "My Sequel" or "MY S-Q-L"): Open source relational database software

    +

    nominatim

    +

    A web service which turns place names and addresses into coordinates so that they can be mapped / associated with an administrative area

    +

    PostgreSQL

    +

    (abbreviated Postgres): Open source object relational database software, a query language similar (but not identical) to that in MySQL is used

    +

    OCR

    +

    Optical Character Recognition: the electronic conversion of scanned images of handwritten, typewritten or printed text into machine-encoded text

    +

    reference data

    +

    data from outside an organisation (often from standards organisations) which is, apart from occasional revisions, static

    +

    scope creep

    +

    When a project's scope slowly and steadily increases, without a formal re-examination of the original terms and requirements

    +

    shapefile

    +

    a file format for geospatial vector data that is primarily used to feed data into mapping applications

    +

    Tilemill

    +

    Hosted and open source mapping software for creating maps

    +

    user story

    +

    short descriptions that outline in plain language the different use cases and possibilities for different types of users for the purposes of determining software requirements

    +

    vector data

    +

    a type of image composed of connected points rather than pixels

    +

    XML

    +

    eXtensible Markup Language: a markup language created to structure, store, and transport data by defining a set of rules for encoding documents in a format that is both human-readable and machine-readable.

    +

    Some of the above entries contain excerpts from Wikipedia.org and the Jargon File v. 4.4.8

    +
    +
    diff --git a/examples/openspending/content/resources/handbook/cover.pdf b/examples/openspending/content/resources/handbook/cover.pdf new file mode 100644 index 00000000..95ac78c6 Binary files /dev/null and b/examples/openspending/content/resources/handbook/cover.pdf differ diff --git a/examples/openspending/content/resources/handbook/images/header.png b/examples/openspending/content/resources/handbook/images/header.png new file mode 100644 index 00000000..d4c9fed1 Binary files /dev/null and b/examples/openspending/content/resources/handbook/images/header.png differ diff --git a/examples/openspending/content/resources/handbook/index.html b/examples/openspending/content/resources/handbook/index.html new file mode 100644 index 00000000..0bf831e0 --- /dev/null +++ b/examples/openspending/content/resources/handbook/index.html @@ -0,0 +1,467 @@ + + + + + + + + + + + + +
    Introduction
    1. Introduction
    2. Working with others
    3. Helping the government help itself

    1. Introduction

    +

    The Spending Data Handbook is addressed to people and organisations who want to use and understand government budgets and spending data in their work. These groups may include government oversight and accountability groups, issue oriented advocacy organisations, journalists covering the latest round of budget cuts or austerity measures, or citizens who just want to inform themselves on what their government is spending. The truth is that for many countries, it's never been easier to access the underlying data relating to government budgets and spending and the expectations have never been higher for governments to release this information in a timely manner. But once you've got the information, making sense of it can be daunting.

    +

    Irrespective of which kind of user is approaching this data, they all have one thing in common: they need to manipulate, analyse and interpret data in their work. Increasingly, we are bombarded with statistics and figures from governments and political parties and it is the job of these organisations to scrutinise and fact-check them, as well as come up with alternative models and solutions. This requires timely, data driven analysis.

    +

    If these groups think that "topic-driven analysis" can drive positive change within governments, there is another group at whom this book is addressed: a strange race of people who self-identify as "civic hackers" and think that technology can be put to work to make government as easy, accessible and effortless for citizens as the internet. While these groups use data processing tools in their day to day lives, their focus is often on enabling others to act on information, rather than having a particular advocacy aim themselves (though there are a few notable exceptions to this rule).

    +
    DON'T PANIC! This book uses a bit of technical vocabulary and terms that may be unfamiliar or used differently in different circles. Anything in italics such as hacker in this paragraph, can be found in the glossary at the back, it may not mean what you think.
    +

    The range of topics tackled by these groups is so diverse that it would be impossible to address all of the questions and issues in one humble book. However, there are some overarching principles and helpful techniques which apply universally to working with government financial data and particularly, how technology might help to do so. This book aims to highlight areas where civic hackers, citizens, Civil Society Organisations (henceforth CSOs), and civil servants working in the field of government transparency could combine forces to achieve common aims. Like the Open Data Handbook (http://opendatahandbook.org/en/), it will be available as a continually evolving, open, educational resource on the internet.

    +

    What we will cover in this book

    +
    • Collaborating with other organizations to pool resources and strengthen your advocacy effort
      • +
    • If you're just starting out, what data to look for and what to ask for (nay, demand!) from your government
      • +
    • The 'Data Pipeline': Tricks and tips for finding, wrangling and systematically processing your data
      • +
    • Getting ambitious, running a technology project
      • +
    • Presenting your findings to engage the public, media and government
      • +
    • Lists and appendices of technical and non-technical resources
      • +

    The Data Processing Pipeline

    +
    +

    How the book was created

    +

    The book was started at a four day book sprint bringing together organisations from around the world from both a technical and a CSO background. Representatives from the Open Knowledge Foundation (UK), Fundar (Mexico), the Centre for Budget and Governance Accountability (India), the Public Interest Advocacy Centre (Bosnia and Herzegovina), and the Sunlight Foundation (US) contributed to the original document. The Book Sprint was facilitated by Adam Hyde, founder of the Book Sprint method and www.booksprints.net.

    +

    Why are we writing this?

    +

    As people who work with this data, we know better than anyone that a few years of a handful groups spreading awareness of spending data to the general public has not been sufficient for enacting earth-shattering change in the budgetary policies and processes in governments around the world. While it's true there's more spending data available than ever before, these efforts have proven to be superficial in some cases and the budget processes themselves have not become much more transparent or participatory.

    +

    Conversely, the armchair auditors that were expected to emerge from the citizenry never really materialised. The notion of citizen auditors and engaging with citizens on an individual and collective basis still struggles to make headway. Clearly, we need to take solutions to this problem to the next level. More groups need to be more involved than ever before.

    +

    What makes this work so tricky?

    +
    • The data people need is not available to the public (and to CSOs).
      • +
    • Alternatively, the data is not available until after all the important decisions within government have been made.
      • +
    • It's difficult to simplify data as complex as budgets and spending and make it accessible to a variety of audiences.
      • +
    • The discourse on budgets and governance in the country is usually replete with jargon and technicalities.
      • +
    • Even when compelling research findings are presented, there's no sense of urgency on the part of key policy actors. Advocacy with a range of stakeholders becomes important.
      • +
    • There is duplication of efforts. One CSO may invest two weeks of its resources into painstakingly cleaning up and extracting data from a policy document published as a PDF while another will be doing exactly the same thing.
      • +
    • Organizations doing this work often lack any kind of peer review process among similar groups.
      • +
    • There may be skills gaps at many CSOs. Skill and knowledge sharing can help all of them achieve common goals.
      • +

    Contribute to the book

    +

    This book is released under a Creative Commons attribution licence, meaning that anyone is free to use and reuse the material provided that it is attributed to the Spending Data Handbook. You can help by doing one of the following:

    +
    • Translate it and customise it for your region. The examples which we include here are the ones which we know best, however, you may feel that in your area, there are topics or examples which would be more relevant. Take the book, remix it and add your own examples.
      • +
    • Correct it and update it - treat it like a Wikipedia article, a living document. The only way that this book will stay relevant and factually correct is with the help of you and other people who know your stuff.
      • +
    • Be inspired - we are filling the book with lots of examples of visualisation and data so inspire you and people you work with.
      • +

    2. Working with others

    +

    We've outlined some ambitious goals for this book, but you don't have to go it alone! In this section we focus on how to work with other organizations to create powerful advocacy efforts, all driven by hard fiscal data! Every CSO, not just government transparency organizations, can leverage the information in this book. Government spending data touches issues covered by a variety of organizations and journalistic outlets, such as human rights, the environment, poverty, scientific research, education, economic policy, and more. But not all these groups may realize it's importance.

    +

    Why work together?

    +

    A common observation has been that CSOs, journalists, and other groups that work with a lot of data encounter a set of similar problems. The problems typically include lack of technical knowledge, duplication of work, lack of wide dissemination of analysis and results, and creating compelling visual products. A lot of these challenges could be mollified if proper channels of communication existed between these organizations.

    +

    One of the most important ways to reduce these technical barriers is to maintain an active dialogue and a process of sharing the methodologies and underlying data in analysis you produce. You can avoid the mistakes of your predecessors by reading their methodologies and learning about idiosyncrasies in the data. Or you may be able to point out mistakes another organization has made. Similarly, if you document how your data has been processed and analyzed, other organizations can benefit (more about that in a few chapters).

    +

    Creating communication can be as easy as joining or maintaining a mailing list of groups that work on similar topics or leverage similar data in their work products. Most webmail services offer free group or mailing list functions, such as Google Groups or Yahoo Groups. If you maintain one, be responsive to messages and active in recruiting members. Email is an extremely low barrier for starting a conversation between organizations.

    +

    If you've already got data products that you want to share, try disseminating them in a couple of different ways. In addition to the methods outlined above, try preparing primers or manuals for other groups that want to start working with datasets that you may be particularly experienced with. You can also organize workshops, meetups, and webinars to familiarize groups with the data you work with. If grassroots issue groups can internalize your data products and present them in a contextually relevant way to their user base, then both groups have gained something from the partnership.

    +

    Another good way to facilitate knowledge sharing and communication between organizations is by conducting internship programmes for interested individuals and organisations. However it is important to keep in mind the target groups. It's great to interface with similar spending data focused groups working at different levels of government than your own, but identifying groups that work on completely different issue areas is important. These groups may feel incapacitated to enact change in the fiscal policy as it relates to their own issues. It then becomes an important job for spending data groups to identify and offer their support. It's especially important to establish strong links with groups that serve those minority populations who are often marginalized in government spending priorities.

    +

    The kind of collaboration discussed above can result in a much broader and stronger coalition of organizations that can advocate for fiscal transparency and detailed disclosure of spending data at all levels of government.

    3. Helping the government help itself

    +

    We've discussed that it's important for CSOs, journalists, and other groups to work together and it's likely that you won't bump up against too much resistance to this idea. But what about when you get pushback from the government even after you've built a strong coalition to advocate for better access to data? Well, one answer you can give them is "Government darling, you're only hurting yourself".

    +

    Whenever the word 'transparency' is mentioned, the first thought that usually springs to mind is a bright light being shone into dark corners of government offices, trying to expose the dark secrets of corrupt bureaucrats and inefficient expenditure. What is often overlooked is that governments also often stand to benefit from more transparent publishing practices. Not only does proactive disclosure help save on the time and cost associated with increased Freedom of Information requests, but other governments within the same nation can benefit from each other as well.

    +

    Connecting different levels of government

    +

    Money at the federal level is often transferred to local governments in the form of grants, direct payments or formula payments. When sub-national governments have better, more timely information on national budgeting priorities, it allows them to adjust their own budgets to account for cuts or increases in certain programs. Similarly, many local governments have small staffs and rely on revenue estimates and models at the federal level to estimate their own revenue.

    +

    This underscores the need for a consistent, standardized information sharing across governments, as it would allow them to share models, best practices, and software with each other, instead of custom building everything from scratch. Furthermore, it would allow for cross-checks on both the federal side and local side for data auditing purposes.

    +

    The next section highlights a couple of case studies for CSOs to use when trying to convince their government to publish better data.

    +

    Case study

    +

    Transparency to combat lag-times for Governments in British Columbia

    +

    When the province of British Columbia built a data portal (http://www.data.gov.bc.ca/), its motivations were primarily:

    +
    • citizen engagement - they wanted citizens to better understand the workings of government
      • +
    • innovation - they wanted people to build applications and tools using the data
      • +
    • making handovers effective - a large number of the workforce were approaching retirement age, and those in charge wanted to make sure that they handed over the necessary information well in advance
      • +

    Probably one of the less-expected impacts was seeing how civil servants themselves used the portal. In 2012 approximately one third of all the traffic originated from government computers. The technology enabled faster access to relevant data within the government departments, contributing to better collaboration on policies that required fiscal data. There was also about 20% increase in the number of Freedom of Information (FoI) requests, showing that releasing a small amount of data fuelled wider interest in data.

    +

    The importance of constant access

    +

    It is somewhat terrifying to think that a lot of budgeting decisions could made almost entirely in the dark. As soon as one executive budget proposal is finalised and published, work often begins on producing the next one. Within governments, those who have to draw up the next year's plan need access (and quickly) to information, such as actual quarterly expenditures, in order to work out whether a government department is properly resourced from the outset, or is drastically under or over spending.

    +

    Case study

    +

    International Aid Transparency Initiative (IATI)

    +

    Building on around 20 years of previous work, the first version of the International Aid Transparency Initiative (IATI) standard was launched in early 2011 and aid donors around the world started publishing to it. The IATI specification documents data about both aid donors and aid activities, enabling comparison and encouraging good practice in data management. The IATI standard also includes space to attach relevant documents and details of project results, to allow the standard to build context around the data and become an end-to-end solution, tracking projects from inception to execution.

    +

    The IATI standard has seen far greater uptake than any previous effort in terms of organisations electing to publish their data in a standard format. These successes are often attributed to the multi-stakeholder nature of the design process, including both policy makers and technical experts. From 2009 to 2010, IATI consulted with a wide range of stakeholders on the design of the technical standard, alongside a parallel process to secure donor support for publishing their aid information.

    +

    In the past it was common to respond to information shortages by building a new database. But by working with open data principles, IATI allows a more distributed solution - where information can flow between organisations in many different ways, not just into a central database.

    +

    Main user groups

    +
    1. Parliamentarians in developing countries gain a better oversight of the aid resources available. Knowing where to allocate resources in their own budgeting processes is vital to ensure that money is spent in the best way. Sometimes, there is transparency-asymmetry between different parts of government. For example, a treasury may be very willing to open up the information it holds, but departments which benefit strongly from aid donations (e.g. departments of health) may be more reluctant to be transparent about aid revenues, as they will not want to 'lose out' from central government budgeting.
      2. +
    2. Donors: Know where their money is going and whether it is being spent correctly.
      3. +
    3. CSOs and private companies who monitor aid effectiveness
      4. +

    The benefits of this approach have already been demonstrated, with many CSOs and charities choosing to follow the IATI Standard, although it was developed primarily with governments in mind.

    +

    Donors publish aid information as a feed which can be read by many different applications, including those created by other donors, by the open data community, and - importantly - by software providers that are developing country financial systems. By providing aid information in a standard format, many different users can access the data in the way they need to - and developing countries can see the resources, which are supposed to be flowing to them.

    +

    Is there anything like this for spending and budgets?

    +

    At present, no. The way many governments report their financial data is a function of how their budget process works or how they use their accounting software. This may not even be consistent within a government, but vary by department and sub-department.

    +

    Could it be useful to create one? We suspect so. The IATI standard for the first time is enabling people to track the money across country boundaries, something which is almost impossible with other types of expenditure, but crucial for those whose job is oversight of government activities and accountability bodies, such as those who work to prevent international money laundering and corruption.

    Data Literacy
    4. Data-driven advocacy and research
    5. Types of data
    6. Getting and cleaning data
    7. Analysis
    8. Using technology in your work

    4. Data-driven advocacy and research

    +

    We are now in a phase where many governments around the world are proactively publishing documents about what they plan to spend (budgets) and actually spend (spending data). Increasingly, this material is available on the internet, so that anybody can access it at any time. Still, too much of the information is released in the form of 'documents' rather than 'data'. Ideally we need both so that inforamtion can be analyzed, re-used and understood. This chapter is a quick overview of some of the raw inputs required for data-driven advocacy and how it works in practice. 

    +

    What do we mean by machine-readable data?

    +

    When we speak about data, what we usually refer to is the notion of machine-readable (http://en.wikipedia.org/wiki/Machine-readable_data) data. Many of the formats most commonly used for policy papers and long-form reports published by most policy-making institutions are PDF files, Word documents, web pages or closed interactive infographics - do not structure information in a way that lends itself to automated analysis and extraction.

    +

    Such documents are formatted for humans (or printers) to interpret, and it can be hard (and in many cases nearly impossible) for a machine to re-construct the elements in the presentation.

    +

    Other formats, such as Excel and CSV files contain a higher level of structured information. For example, in an Excel file you can mark a number of cells and easily calculate their sum. Even more exotic and useful file formats, such as XML documents, JSON APIs or Shapefiles may not have easy-to-use viewer applications. You can think of them as the glue that connects different systems on the web, so that different databases can work together in a seamless fashion.

    +

    Why do CSOs need it?

    +

    What asking for machine-readable bulk data means for CSOs is simple: you won't have to spend a lot of time manually extracting data from reports into spreadsheets to be able to filter, sort and analyse it - a process which is both time-consuming and can introduce errors.

    +

    What to ask for when asking for data: a checklist

    +

    In the next section 'Getting Data' - we will deal with asking governments for data (or getting it via other means). To set the scene for this and to work out whether your government actually publishes usable data already, have a quick look at the following questions:

    +
    • Is the government's data published in a machine-readable format? E.g. CSV, XML, JSON. While there is nothing wrong with publishing a PDF to support a data release (in fact it is often nice to have a nicely-laid out document to cross reference and sanity-check data) it shouldn't be the only thing published and if you are asking for a policy document, ask for the underlying data in a spreadsheet so you can check the numbers.
      • +
    • Does the government publish a 'data dictionary' to explain the terms used in the dataset? This should include definitions of column headers, explanations of terms and ranges used within the main body of the data, explanations of any changes in terminology which have been introduced since last time the dataset was released
      • +
    • How is the data that is being published actually used internally by governments? Do some sanity checks on the minimum and maximum values of different columns to make sure they fall into the documented ranges and don't seem out of place. Do you see negative values when you don't think you should? Negative values usually mean money owed.
      • +
    • Is the structure of the data the same across years? If not is there a description of how it changes? It never hurts to contact the publisher and ask questions about the change and why it occurred. The publisher may have their name and contact details on the report or webpage. If there is no named contact then call the department's enquires number or send a message to their email address asking to meet or discuss the data.
      • +
    • How aggregated is the data? What is the number of real-world financial transactions that are expressed by a single line of the dataset you have? For budgets this will mostly be hard to tell - but with transactional expenditure you want to make sure that the data is fairly disaggregated. Ideally, each entry represents a transaction - but even if this isn't true you'll still want to ensure the number is not in the tens or hundreds of thousands (e.g. government programmes as a whole).
      • +
    • Ask for reference data. If your budget or spending data is augmented with reference data, make sure you have access to it. This might include functional or category codes on budget line items, location codes for describing recipient location, or codes that indicate the status of the record. 
      • +
    • Ask also for the guidelines people were given when creating the dataset. This will make it easier to understand what is included within the data, e.g. are the numbers in thousands / millions. 
      • +
    • Final tip: if the data you want is not given then narrow your scope. Your chances of success will be higher if you narrow the scope of the data you're requesting from the government and you are specific. Government is the de facto keeper of all kinds of data, so parameters that narrow your request are always helpful.
      • +

    An introduction to data-driven advocacy

    +

    Is going out and provoking a riot the best way to get a Government to take onboard your message? There are alternatives: hit them with the data hammer instead!

    +

    Making evidence-based policy proposals consists from three major phases: formulating your assumption, analysis (which often leads to re-formulating your assumption, and presenting your data in an engaging way in a policy proposal.

    +

    Analysing assumptions

    +

    Asking the right question is key to getting the most out of your data. We all make assumptions, and our organisation may have a particular standpoint on a given issue. Our first task is always to formulate our assumptions and then interrogate them ferociously. Although we try to be rational in this process, our judgement is often influenced by our subjective goals, values, and beliefs. Sometimes, you'll need to revisit your assumptions several times over to ensure they are valid and you can back them up with data. Once you know your policy problem is definitely a problem, you can work to package it in a way that's appropriate for your target audience. 

    +

    What is public interest? 

    +

    Often our job is to act in the public interest by analysing conflicting assumptions and working out which one is more valid. For example, in Greece, Spain, and many other European countries people protest almost everyday as the Government cuts spending to bring down its budget deficit. If the Goverment wanted to keep its current level of spending, but increased taxes to increase its revenue, different citizens groups would still protest depending on which taxes are to be increased. In any case, there will always be more than one interpretation of any Government policy, and interested side to support it, or not.

    +

    Policy analysis

    +

    Once we have a well defined policy problem, specific goals, or results different stakeholders are trying to achieve, and corresponding instruments they are using in this process, we may systematically search for the specific data needed to create our own policy proposals. This data can be obtained either from the Government, some other sources e.g. academic journals, private companies, or generated by ourselves. When data is gathered we will use a specific methodology to analyze it, and based on this analysis we will approve or reject our assumptions. If the assumption is rejected, based on our findings we will have to make the new assumption, and start the process from the beginning. If our assumption is approved, we will use our results to make a policy proposal to the Government.

    +

    Policy proposals

    +

    For CSOs it is important to recognize who is a decision maker, hence, who you should be targeting with your policy proposal. Policy proposals should be methodologically well structured, evidence-based, open for debate, and scientificaly evaluated. Governments will seldom take our policy proposals as their own policymaking, but may actually change its course of action, get new insights, views, and understanding of the subject. We may also use policy briefs to approach Government officials, or press releases to get the attention of the public. 

    +

    +

    Case study 

    +

    Fish subsidies

    +

    The influence CSOs have on government policy comes from a wide and varied set of activities. These can range from producing a widely shared dataset or infographic which subtly influences the mood of policy makers, to more targeted CSO advocacy and lobbying on issues they are experts.

    +

    The Fish Subsidies group (http://fishsubsidy.org) are a nice example of a CSO engaged in targeted activites. Having collected a comprehensive set of data on Fishing subsidies paid under the European Union’s common fisheries policy and they break this down into payments for every EU member state, and then complemented this with activites of fishing. They have produced a report (http://is.gd/XYPgq5) assessing the environmental and social impacts of the Financial Instrument for Fisheries Guidance between 2000 and 2006. This extensive document fed directly into the EU political decision making process.  

    5. Types of data

    +

    For the purposes of this manual, we have decided to abstract the meaning of the terms "budget data" and "expenditure data" to fit into a broader context. While these terms may have different meanings on a country by country basis, they are intended to be taken as defined in this section, throughout the rest of the manual. In this section, we look briefly at the two different types of data and what questions can be addressed using them.

    +

    +

    Budget data is defined as data relating to the broad funding priorities set forth by a government, often highly aggregated or grouped by goals at a particular agency or ministry. For instance, a government may pass a budget which contains elements such as "Allocate $20 million in funding for clean energy grants" or "Allocate $5 billion for space exploration on Mars". These data are often produced by a parliament or legislature, on an annual or semi-annual basis.

    +

    Spending data is defined as data relating to the specific expenditure of funds from the government. This may take the form of a contract, loan, refundable tax credit, pension fund payments, or payments from other retirement assistance programs and government medical insurance programs. In the context of our previous examples, spending data examples might be a $5,000 grant to Johnson's Wind Farm for providing renewable wind energy, or a contract for $750,000 to Boeing to build Mars rover component parts. Spending data is often transactional in nature, specifying a recipient, amount, and funding agency or ministry. Sometimes, when the payments are to individuals or there are privacy concerns, the data are aggregated by geographic location or fiscal year.

    +

    The fiscal data of some governments may blur the lines of these definitions, but the aim is to separate the political documents from the raw output of government activity. It will always be an ultimate goal to link these two datasets, and to allow the public to see if the funding priorities set by one part of the government are being carried out by another part, but this is often impractical in larger governments since definitions of programs and goals can be "fuzzy" and vary from year to year.

    +

    +

    Budget data

    +

    Using the definitions above, budget data is often comprised of two main portions: revenue and taxation data and planned expenditures. Revenue and spending are two sides of the same coin and thus deserve to be jointly considered when budget data is released by a government. Especially since revenue tends to be aggregated to protect the privacy of individual taxpayers, it makes more sense to view it alongside the budget data. It often appears aggregated by income bracket (for personal taxes) or by industrial classification (for corporate taxes) but does not appear at all in spending data. Therefore, budget data ends up being the only source for determining trends and changes in revenue data.

    +

    Somewhat non-intuitively, revenue data itself can include expenditures as well. When a particular entity or economic behaviour would normally be taxed but an exception is written into the law, this is often referred to as a tax expenditure. Tax expenditures are often reported separately from the budget, often in different documents or at a different time. This often stems from the fact that they are released by separate bodies, such as executive agencies or ministries that are responsible for taxation, instead of the legislature (http://internationalbudget.org/wp-content/uploads/Looking-Beyond-the-Budget-2-Tax-Expenditures.pdf).

    +

    Budgets as datasets

    +

    A growing number of governments make their budget expenditure data available as machine-readable spreadsheets. This is the preferred method for many users, as it is accessible and requires few software skills to get started. Other countries release longer reports that discuss budget priorities as a narrative. Some countries do something in between where they release reports that contain tables, but that are published in PDF and other formats from which the data is difficult to extract.

    +

    On the revenue side, the picture is considerably bleaker, as many governments are still entrenched in the mindset of releasing revenue estimates as large reports that are mostly narrative with little easily extractable data. Tax expenditure reports often suffer from these same problems.

    +

    Still, some areas that relate to government revenue are beginning to be much better documented and databases are beginning to be established. This includes budget support through development aid, for which data is published under the IATI (http://www.aidtransparency.net/) and OECD DAC CRS (http://stats.oecd.org/Index.aspx?DatasetCode=CRSNEW) schemes. Data about revenues from extractive industries is starting to be covered under the EITI (http://eiti.org/) with the US and various other regions introducing new rules for mandatory and granular disclosure of extractives revenue. Data regarding loans and debt is fairly scattered, with the World Bank providing a positive example (https://finances.worldbank.org/), while other major lenders (such as the IMF) only report highly aggregated figures. An overview of related data sources can be found at the Public Debt Management Network (http://www.publicdebtnet.org/public/Statistics/).

    +

    Connecting revenues and spending

    +

    It is highly desirable to be able to determine the flow of money from revenues to spending. For the most part, many taxes go into a general fund and many expenditures come out of that general fund, making this comparison moot. But in some cases, in many countries, there are taxes on certain behaviours that are used to fund specific items.

    +

    For example, a car registration fee might be used to fund the construction of roads and highways. This would be an example of a user fee, where the main users of the government service are funding it directly. Or you might have a tax on cigarettes and alcohol that funds healthcare grants. In this case, the tax is being used to offset the added healthcare expense of individuals taking part in at-risk activities. Allowing citizens to view what activities are taxed in order to pay for other expenditures makes it possible to see when a particular activity is being cross-subsidized or heavily funded by non-beneficiaries. It can also allow them to see when funds are being diverted or misused. This may not always be practical at the country level, as federal governments tend to make much larger use of the general fund than other local governments. Typically, local governments are more comprehensive with regards to releasing budget data by fund. Having granular, fund-level data is what makes this kind of comparison and oversight possible.

    +

    +

    What questions can be answered using budget data?

    +

    Budget expenditure data has an array of different applications, but it's prime role is to communicate to it's user broad trends and priorities in government spending. While it can help to have a prose accompaniment, the data itself promotes a more clear-cut interpretation of proposed government spending over political rhetoric. Additionally, it is much easier to communicate budget priorities by economic sector or category than it is at the spending data level. These data also help citizens and CSOs track government spending year over year, provided that the classification of the budget expenditure data stays relatively consistent.

    +

    Spending data

    +

    For most purposes, spending data can be interpreted as transactional or near-transactional data. Rather than communicating the broad spending priorities of the government like budget data should, spending data is intended to convey specific recipients, geographic locations of spending, more detailed categorization, or even spending by account number.

    +

    Spending data is often created at the executive level, as opposed to legislative, and should be more frequently reported than budget data. It can include many different types of expenditures, such as contracts, grants, loan payments, direct payments for income assistance and maintenance, pension payments, employee salaries and benefits, intergovernmental transfers, insurance payments, and more.

    +

    Some types of spending data - such as contracts and grants - can be connected to related procurement information (such as the tender documents and contracts) to add more context regarding the individual payments and to get a clearer picture of the goods and services covered under these transactions.

    +

    Opening the checkbook

    +

    In the past five years, there have been a spate of countries and local governments that have opened up spending data, often referred to as "checkbook level" data. These countries include, but are not limited to, the US (including various state governments), UK, Brazil, India (including some state governments) and many funds of the European Union.

    +

    Disclosure thresholds

    +

    At least two of these countries have imposed seemingly arbitrary thresholds on the size of transactions that are included. For example, the US and the UK exclude transactions under $25,000 and 25,000 GBP, respectively. Are these thresholds appropriate? That can't be known for sure without more information about how these numbers were arrived at. Principally, having thresholds or exceptions to the reporting of this data depends on the underlying systems that drive disclosure of this data. Are these systems linked directly with the accounting systems already used in the government, easing the burden of disclosure? If so, the threshold for excluding transactions should be very low (setting aside for a moment the cases that require redaction for privacy purposes).

    +

    If the systems are mostly divorced, as is the case with the US, then it begs the question, why? The more steps and processes between the internal government accounting systems and the public accounting systems, the higher the chance of error in the data and chance for omission of data. It also undermines the primary goal of public oversight if there are separate systems. However, governments often struggle with IT resources and contracting, which presents a tension between releasing any spending data at all, and release that is consistent with the above principles. If a threshold is necessary, then the amount should be consistent in size and scope with the overall expenditure level for that particular government. It is not appropriate, for example, that the threshold for the US State of Maryland spending reporting is also $25,000, when their annual budget is only a fraction of the federal government's budget.

    +

    Release early, release often

    +

    Spending data should be released in a relatively timely fashion, at least a monthly or quarterly basis. The timeliness of this data is what allows users to see if the spending priorities in the budget data are being reflected in the spending data. Also it allows the public and government stakeholders to view the current year's spending on a more detailed level as the next year's budget is being decided.

    +

    A good example of such release of spending information can be the Indian experience, especially the Employment Guarantee Programme, one of the major National flagship programmes on providing demand-based employment to the rural working age-group population in India. Its Management Information system (MIS) has become the most effective way of getting information on spending on a monthly basis. The data is updated monthly in an accessible spreadsheet format (Excel) at the sub-national government level. This makes the data transparent and available in the public domain to be equally accessed by all. The village level household database has internal checks for ensuring consistency and conformity to normative processes. It includes separate pages for approximately 250,000 local governments at the village level, 6,465 Blocks, 619 Districts and 34 States & Union Territories. The portal places complete transaction level data in the public domain.

    +

    However, problems related to maintaining an MIS in every state in a functional form and releasing continuous flow of data have been contentious issues. The major concerns emerge from lack in technical capacity as well as cost related issues. A cumbersome back-end system for supplying the data requires installation of a specific software with prerequisite configurations and technical operators with specific capacities. These requirements have raised costs and put a great demand on technology to ensure a continuous flow of data on the programme, specifically in most interior parts of the country and hence affect timely release of data.

    +

    What questions can be answered using spending data?

    +

    Spending data can be used in several different areas: oversight and accountability, strategic resource deployment by local governments and charities, and economic research. However, it is first and foremost a primary right of citizens to view detailed information about how their tax dollars are spent. Tracking who gets the money and how it's used is how citizens can detect preferential treatment to certain recipients that may be illegal, or if certain political districts might be getting more than their fair share.

    +

    It can also help local governments and charities respond to areas of social need without duplicating federal spending that is already occurring in a certain district or going to a particular organization. Lastly, businesses can see where the government is making infrastructure improvements and investments and use that criteria when selecting future sites of business locations. These are only a few examples of the potential uses of spending data. It's no coincidence that it has ended up in a variety of commercial and non-commercial software products -- it has a real, economic value as well as an intangible value as a societal good and anti-corruption measure.

    6. Getting and cleaning data

    +

    As you look into a new, data-driven project, you'll find it easy to imagine all the analysis and comparisons that could be done to prove a particular point or to explore some area of government activity. As you then look into the availability of data on a particular topic, disillusionment will quickly set in: information may be hard to find, unstructured or just not available to the public at all. As you search for data, there are a number of possible techniques you may want to investigate. Some are based on the governments cooperation while others use technology to bridge the gap between how information is shared and what is necessary for a data-driven advocacy project.

    +

    Getting data

    +
    Data processing pipeline
    +

    An increasing number of governments have accepted the need for pro-active transparency. When releasing open data, governments commit to making machine-readable information available on the web, in a form that enables anyone to use, re-use and re-distribute the data without legal or technical restrictions. Such releases should happen in a timely manner and include detailed, disaggregated data. Many countries have set up dedicated open data portals which provide easy access to all published datasets and relevant metadata (i.e. information about when a dataset was last updated, who published it and where documentation regarding format and contents can be found). The benefit of using open data when it's available includes the ease of access, but also usually a more authoritative base for any further analysis.

    +

    In some cases, open data is provided through an application programming interface (API), often a web-based method for retrieving, searching or even updating the available information dynamically. APIs provide up-to-date data in a granular and filtered form, removing the need to repeatedly process and update source files.

    +

    A common use case for APIs is relatively time-sensitive information, such as procurement calls and contracts which are released every day. In the UK, BusinessLink (http://www.contractsfinder.businesslink.gov.uk/data-feed.aspx) provides a number of data feeds which contain information about procurement notices. Similarly, the USASpending portal provides a set of APIs that can be used to retrieve up-to-date grants information for the US federal government (http://usaspending.gov/data).

    +
    If you are looking for a list of data catalogs from around the world, DataCatalogs.org is curated by experts in this area from around the world. Search for your country and see what you find.
    +
    +

    Freedom of information

    +

    Even before the rise of open data, many countries decided to increase the transparency of their governments by introducing freedom of information (FoI) legislation. Such laws enable every citizen to request documents and other material from parts of the government which do not merit special protection (e.g. due to concerns over privacy, national security or commercial confidentiality).

    +

    Journalists, activists and CSOs have long had channels of acquiring information. Sometimes, having a good relationship with a press-officer or a civil servant is good enough and making a formal request for information is unnecessary (your friendly press-officer may even feel slightly offended if you don't ask them nicely first). FoIs generate a lot of paperwork (hence grumpy civil servants), so if you do have the contacts, it may be a good idea to ask nicely first!

    +

    Freedom of Information requests often require some degree of preparation, so that the documents or databases that are requested are clearly identified, you know which department or unit is in charge of it and you can address possible concerns over privacy or commercial confidentiality in your request.

    +

    While freedom of information legislation is in force in many countries, it was often made before the need for structured data became apparent - thus many laws do not allow the citizen to specify a particular format. Many governments choose to release information on paper rather than in a structured, digital form, making the data processing step more painful. Still, the legally binding character of freedom of information requests often makes them an invaluable tool in the process of gaining access to financial data.

    +

    FoI requests may be necessary when you want to get more detail on the projects that government money is funding. Often the transactional spending data released will include only a brief description of the project, if at all. To get more information about it, you might need to submit an FoI request. For instance, if you have the high level payment information for a contract that includes the recipient, location and total amount, but you want to know the details of the contract deliverables, you will probably need to submit an FoI request for the full contract.

    +

    A good example of this process is the Sunlight Foundation's request for information on the Airport Improvement Program in the United States. The program accepts applications from airports around the country for infrastructure improvement grants, such as repaving a runway. Each project is assigned a safety priority rating and is prioritized in a queue. The high level spending information for this program was available in USASpending.gov, but since the priority ratings are specific to this program and not spending data in general, they were not included in that dataset. The Sunlight Foundation submitted a FoI request for the full dataset, including the priority ratings. After that, they were able to determine when airports with low priority projects were getting money, and how often. So the lesson is, if you see some interesting patterns in your high level spending data, don't be afraid to dig deeper and ask for more detailed program information.

    +

    Wanting to submit a request, but not sure where to start, who to address your request to or how to write it? Access Info (http://www.access-info.org/) are an organisation who work to help people obtain the information they require from the public bodies that hold it. They have also produced a toolkit (http://www.legalleaks.info/toolkit.html) to using FoIs. It's primarily aimed at Journalists, but most of the tips are equally relevant for CSOs.

    +

    Data scraping

    +

    Unlike open data or freedom of information requests, data scraping does not rely on the cooperation of government authorities for acquiring machine-readable documents. Scraping refers to the process of transforming unstructured documents - online database interfaces, PDF files or even printed documents into a more structured form (see section below on Optical Character Recognition for more tips on how to do this). Many tools and techniques enable such refinement by re-interpreting documents generated for humans into a more structured form which can be analysed, aggregated and distributed in new ways.

    +

    While there is an increasing number of easy-to-use scraping tools which do not require much technical knowledge, more complex processes - such as the automated scraping of thousands or millions of web sites or the mass interpretation of PDF files require some programming so that the process can be fully automated and produce reliable results.

    +

    In some cases, the only way to gain access to a set of figures is through the digitization of printed material. While scanners and optical character recognition (OCR) software can be used to import such documents, the high cost and low data quality generated through this approach often it an unattractive one.

    +

    A very accessible guide to scraping has been published (https://leanpub.com/scrapingforjournalists). Again, it brands itself at journalists (they're getting a little spoiled aren't they?), but scraping is scraping and so a worthwhile read for CSOs.

    +

    Another great source for tutorials, as well as being a tool itself to help with scraping is ScraperWiki. Using Scraperwiki (https://scraperwiki.com/) has the additional benefit of anything that you scrape being available to others, as most of the scraped data goes into a public data store.

    +

     

    +

    Getting data out of scanned documents

    +

    When you deal with scanned documents, the crucial step in the extraction process is to have the computer attempt to recognize any characters - letters, numbers and other signs. Optical character recognition (OCR) software is built to do this, accepting scanned pictures and PDF documents as an input.

    +

    There are both commercial software products for OCR (such as ABBYY FineReader, http://finereader.abbyy.com/), and some open-source software packages, such as Google's Tesseract (http://code.google.com/p/tesseract-ocr/). In general, the quality of all automatic recognition is limited, and you should make sure to cross check any numbers coming from scanned material against the printed documents.

    +

    Keeping the data around

    +

    As you retrieve data from the government (or other sources), it's easy to just consider the websites it has been released on as a permanent resource. Still, experience has shown that data does go away: whether it is through government re-designing its web sites, new policies that retract transparency rules or simple system failures.

    +

    At the same time, downloading complete copies of web sites - a process called mirroring - is a fairly well-established technique that can easily be deployed by civil society organisations. Mirroring involves an automated computer program (for a list see: http://en.wikipedia.org/wiki/Web_crawler) harvesting all the web pages from a specified web page, e.g. a ministry home page. In most cases, it is also possible to find old versions of web sites via the Internet Archive's Wayback machine (http://archive.org/web/web.php), a project that aims to create up-to-date copies of all public web sites and archive them forever.

    +

    Cleaning data

    +
    +

    Data Processing Pipeline

    +

    As you acquire government spending information, you will notice that such data often has many inconsistencies: program names are used inconsistently, financial amounts will be stated in badly formatted numbers, while some data may not be usable at all due to file corruptions. In short: data always needs to be cleaned and processed. In fact, processing, augmenting and cleaning the data is very likely to be the most time- and labour-intensive aspect of your project.

    +

    Processing stages for data projects

    +

    While there are many different types of data, almost all processing can be expressed as a set of incremental stages. The most common stages include data acquisition, extraction, cleaning, transformation, integration, analysis and presentation. Of course, with many smaller projects, not each of these stages may be necessary.

    +

    In this process, each stage fulfils a unique purpose

    +
    • Acquisition describes gaining access to data, either through any of the methods mentioned above or by generating fresh data, e.g through a survey or observations.
      • +
    • In the extraction stage, data is converted from whatever input format has been acquired (e.g. XLS files, PDFs or even plain text documents) into a form that can be used for further processing and analysis. This often involves loading data into a database system, such as MySQL or PostgreSQL.
      • +
    • Cleaning and transforming the data often involves removing invalid records and translating all the columns to use a sane set of values. You may also combine two different datasets into a single table, remove duplicate entries or apply any number of other normalizations.
      • +
    • Analysis of data to answer particular questions we will not describe in detail in the following chapters of this book. We presume that you are already the experts in working with your data and using e.g. economic models to answer your questions. The aspects of analysis which we do hope to cover here are automated and large-scale analysis, showing tips and tricks for getting and using data, and having a machine do a lot of the work, for example: network analysis or natural language processing.
      • +
    • Presentation data only has impact when it is packaged in an appropriate way for the audiences it needs to aim at.
      • +

    As you model a data pipeline, it is important to take care that each step is well documented, granular and - if at all possible - automated. This is particularly true when processing more complex datasets, such as transactional government expenditure which has been scraped from a government website.

    +

    Data provenance

    +

    Good documentation on data provenance (the origin and history of a dataset) can be compared to the chain of custody which is maintained for criminal investigations: each previous owner of a dataset must be identified, and they are held accountable for the processing and cleaning operations they have performed on the data. For Excel spreadsheets this would include writing down the steps taken in transforming the data, while advanced data tools (such as Open Refine, formerly Google Refine), often provide methods of exporting machine-readable data containing processing history. Any programs that have been written to process the data should be available when users access your end result and shared as open-source code on a public code sharing site such as GitHub.

    +

    Tools for documenting your data work

    +

    Documenting the transformations you perform on your data can be as simple as a detailed prose explanation and a series of spreadsheets that represent key, intermediate steps. But there are also a few products out there that are specifically geared towards helping you do this. Socrata is one platform that helps you perform transforms on spreadsheet-like data and share them with others easily. You can also use the Data Hub (pictured below), an open source platform that allows for several versions of a spreadsheet to be collected together into one dataset, and also auto-generates an API to boot.

    +
    +

    Normalizing data

    +

    Data that comes from the government is often generated across multiple departments by hand. This can result in inconsistencies in what kinds of values or formats are used to describe the same meaning. Normalizing values to be consistent across a dataset is therefore a common activity.

    +

    First, you want to start by finding all of the distinct ranges of values for the different columns in your dataset. You can accomplish this by using a database query language (such as SQL's DISTINCT), or by simply using the 'filter' property on a spreadsheet program.

    +

    For example, if you have a spreadsheet with contracting data, and one column is 'Competed?', you would expect the values to be 'yes' or 'no'. But if this spreadsheet is an amalgam of spreadsheet data from multiple users and departments, your values could vary among the following: 'Y', 'YES', 'yes', 1, 'True', 'T', 't', 'N', 'NO', 'no', 0, 'False', 'F', 'f', etc. Limiting all of these potential values to two clear options will make it easier to analyse the data, and also easier for those who follow in your footsteps.

    +

    Especially with financial data, numbers can be formatted several different ways. For example, are your negative values represented with a '-' or placed inside '( )' or possibly even highlighted in red? Not all of these values will be easily read by a computer program (especially the color), so you'll want to pick something clear and consistent to convert all your negative values to (probably the negative sign).

    +

    Is all your numerical data measured out in ones or is abbreviated in thousands? Especially with budget data, order of magnitude errors are not uncommon when one department thinks they're reporting in thousands or millions by default but others expand their data all the way to the ones place. Are some values in scientific notation (e.g. 10e3938)? Make sure all your values are consistent, otherwise your analysis could contain serious errors.

    +
    A column of data requiring name normalization
    +

    Data quality

    +

    Data cleaning and data quality go hand in hand. You may find it helpful to conduct data quality tests after your analysis in order to measure your confidence in the data, or before to decide whether you want to do the analysis at all. Data quality can mean a few different things but here or some key factors to consider.

    +

    Completeness

    +

    Is all your data there? If you have a time series of some data, do you have approximately the same number of rows for each year or are there major outliers? An extremely high or low number of rows that looks out of place may indicate duplicate or lost rows. Or it could indicate a change in how data was reported at that time. Are there values for each column where values are required (according to the documentation)?

    +

    Accuracy

    +

    Does your dataset match up with other available data out there? This can be difficult to confirm, but it is often useful to employ a sampling methodology. Select a significant sample of your records (3-5%) and decide which values you can check for accuracy. If your data is transactional contract data, you could request the full contracts for your sample records and compare the structured data you have with the prose of the contract to confirm that it is correct. You can sometimes check contract and grant amounts from journalistic outlets and even company websites.

    +

    Timeliness

    +

    Sometimes governments can lag behind in their reporting. Even if a dataset says it's for a specific time period, do you notice a long tail of records closer to the end of the time frame? If so, you may want to restrict your analysis to the subset of data where you have a healthy sampling.

    +

    Case studies

    +

    Some projects have gone beyond simple assessments of data quality to ensure confidence in the underlying result of a project, but are projects completely about data quality themselves!

    +

    The Open Knowledge Foundation recently created a data quality monitor for UK transactional expenditure data (http://openspending.org/resources/gb-spending) which has been developed in cooperation with the data.gov.uk team. Each imported file is documented, highlighting any mistakes in the data structure or formal representation of the data. More information at:

    +

    The Sunlight Foundation has run a yearly report called Clearspending (http://sunlightfoundation.com/clearspending/) for the past three years that assesses the quality of the data reported in the US federal spending dashboard, USASpending.gov. The results haven't been very good but have highlighted the need for data standardization and oversight of the data release process.

    +

    As an even more thorough technique, little programs and tests can be written to check data validity along the processing pipeline by testing certain assumptions about each record in the dataset. If you do not have the option of creating such tests, you may still be able to create a spreadsheet in which you record known defects and issues with the data, including source files that did not open and inconsistent column values.

    +

    The most powerful data quality tool is much more simple, though: sharing the data. Without access to your source data, others will have to trust your word blindly. Therefore, sharing both the source and processed forms of your dataset is essential to discussing data quality.

    +

    Categorization and reference data

    +

    One of the most powerful ways of making data more meaningful for analysis is to combine it with reference data and code sheets. Unlike transaction data - such as statistical time series or budget figures - reference data does not describe observations about reality - it merely contains additional details on category schemes, government programmes, persons, companies or geographies mentioned in the data.

    +

    For example, in the German federal budget, each line item is identified through an eleven-digit code. This code includes three-digit identifiers for the functional and economic purpose of the allocation. By extending the budget data with the titles and descriptions of each economic and functional taxonomy entry, two additional dimensions become available that enable queries such as the overall pension commitments of the government, or the sum of all programmes with defence functions.

    +

    The main groups of reference data that are used with government finance include code sheets, geographic identifiers and identifiers for companies and other organizations:

    +

    Classification reference data

    +

    Reference data are dictionaries for the categorizations included in a financial datasets. They may include descriptions of government programmes, economic, functional or institutional classification schemes, charts of account and many other types of schemes used to classify and allocate expenditure.

    +

    Some such schemes are also standardized beyond individual countries, such as the UN's classification of functions of government (COFOG) and the OECD DAC Sector codes (http://www.oecd.org/dac/aidstatistics/dacandcrscodelists.htm). Still, the large majority of governments use their own code sheets to allocate and classify expenditure. In such cases, it is often advisable to request access to the code list versions used internally by government, including revisions over time that may have changed how certain programmes were classified.

    +

    A library of reference data that can be re-used across different projects and it is a valuable asset for any organization working with government finance. Sharing such data with others is crucial, as it will help to enable comparable outputs and open up options for future data integration. Existing repositories include the IATI Standard (http://iatistandard.org/) and datahub.io.

    +

    Geographic identifiers

    +

    Geographic identifiers are used to describe administrative boundaries or specific locations identified in a dataset. While some regional classifications (such as the EU NUTS) are released on the web, there is also an increasing number of open databases which contain geographic names - including geonames.org and the recently developed world.db.

    +

    Another related technique is the process of reverse geo-coding: translating a human-readable address into a pair of coordinates. Services like nominatim (http://nominatim.openstreetmap.org/) will not only enable users to generate precise maps of projects in a region, they will also return the responsible administrative boundary for many coordinates. This means that projects which are given by precise address can also be aggregated by state, region or any other geographic unit.

    +

    Additionally, many countries have shapefiles of their political and geographic districts available (usually through the census or interior bureaus) that can be imported into custom mapping applications, like TileMill (http://mapbox.com/tilemill/).

    +

    Company and organisational identifiers

    +

    As you look into spending data that includes recipients outside the government, you'll find companies which act as suppliers to government, but also other types of entities including charities, associations, foreign governments, political parties and even individuals which act as recipients of direct assistance.

    +

    Identifying such entities is notoriously hard, since the only information kept by government is often a simple name (which may not uniquely identify the beneficiary, for example "MS"). While most (if not all) countries maintain company registers which assign some type of unique identifier to a company, these databases are often not accessible in bulk and not used coherently across different parts of government. Alternative identifiers - such as tax identifiers and company IDs from private business information suppliers (such as Dun & Bradstreet in the US) - further complicate this process.

    +

    As an alternative, open registries are beginning to compile organisational identifiers in a form that is easy to re-use and thus enables the sharing of databases which have been augmented with such information. OpenCorporates.com (http://opencorporates.com) is a startup that collects information from companies world-wide and provides a convenient API to match datasets with the list of known countries. The IATI project for aid transparency is working towards similar standards for other organisations, such as foreign governments and charities active in the development space.

    7. Analysis

    +

    +

    Data Processing Pipeline

    +

     

    +
    Dear reader - we know you are the experts. We're not going to teach you all the basics, but this section's aim is to show you how technology can help speed up, scale up or simplify your working practices. For this reason in this section we will say only a few words about statistical methods, even though many of you will be very familiar with those.
    +

    Once you have your data in a suitable format to be dealt with, much of your analysis can be automated with a custom program or inside spreadsheet software. Obviously the deeper your data dive is, the less automated things become, but having a standard set of post-processing, analytical steps to run on a dataset helps make sure you haven't missed anything. Some of the steps may seem basic, but it always helps to have a checklist!

    +

    Sorting data

    +

    Possibly the simplest way to find some interesting points is to sort by a column or category. In spending datasets, it's common to rank data by the largest transactions and look at the beneficiaries. This also gets you the maximum or minimum values of a column, which could be interesting.

    +

    Basic statistical analysis

    +

    Statistical analysis such as calculating percentiles or the average of a dataset, are essential to characterize your data. These are available in most programming languages as well as most spreadsheet programs like excel. A more complete list of statistical tools and resources can be found at the resources chapter.

    +
    • An average is a measure of central tendency in a sample. It can be described as the "typical" or representative value of a data set. Knowing the average helps detect whether any single payment is atypical.
      • +
    • A percentile is the value of a variable under which a certain percentage of the data fall. For example, if a transaction amount is in the 90th percentile, then it is larger than 89% of the transactions in the dataset.
      • +
    • A standard deviation measures how far away a value is from the average. If your standard deviation is low, then it means your row values are clustered together. If it's high, then they are spread out. By checking for values that are one, two, or three standard deviations outside the mean, you can detect outliers. Outliers could be errors or extremely interesting data points.
      • +

    Summarizing or aggregating by categories

    +

    Sometimes data is available at a very high level of detail, making it hard to see the big picture. It's a good idea to have a standard set of categories to aggregate by, such as geographic unit, functional classification, account number, fiscal year or other temporal unit, government department, or even recipient.

    +

    Normalization for comparability

    +

    In order to be able to compare data across time or other parameters, you may need to systematically adjust the data values. Here are some examples.

    +
    • Deflation or inflation. Expenditure data is naturally obtained with current (or nominal) money value, but such data is not directly comparable across different years since inflation affects the value of money. In order to compare currency across years, it is necessary to deflate or inflate the value according to a base year. For example if we wish to compare the budget of a program in dollars from 2000 with the same program in 2012 we must adjust the value of the 2000 data and the 2012 dollars to a base year. Deflator and inflator factors are calculated by central banks in most countries and a comprehensive list by country can be found at the World Bank website (http://worldbank.org)
      • +
    • Scale and normalize values. When we wish to compare patterns of budget programs sometimes the difference in scale between budgets may prove difficult to display in a graphic form. In such cases is useful to compare percentages instead of absolute numbers. For instance, it is very common to express budget and spending data as percentages of a country's GDP, instead of in absolute numbers. If you're mapping spending, make sure to scale your data by population density. Otherwise, your map will most likely just be a map of where your populace lives.
      • +
    You may want to visualize your data to get better understanding of your initial assumptions. This may save you a lot of time in case your initial assumptions were not good, and may also help you to get more insight into new findings your data might contain but previously were not clearly visible. For this purpose you can use scatter plots, line graphs, heat maps, bar charts.

    8. Using technology in your work

    +

    As you move into more advanced data projects, you may find yourself in a situation where a research project turns into an effort that involves coders, designing databases and web sites. It is important that you take a step back and realize that you are now running not just an advocacy effort, but also an IT project.

    +

    There are many difficulties which CSOs face when developing software. Some common issues include:

    +
    • Difficulties in finding qualified developers that want to contribute to your projects at a reasonable rate, as well as in the communication between CSO staff and developers.
      • +
    • Clearly communicating the requirements for the software so that both non-technical and technical staff share a vision for the outcome of the project.
      • +
    • The estimation of time and resources for particular tasks, especially how to handle projects that drastically overrun the timeframe and funding they were initially assigned.
      • +
    • Evaluating the work of developers to ensure that the product that has been delivered is according to what has been agreed, especially in small projects with only a single coder or when working with external contractors.
      • +
    • Maintaining the project after the main development period has finished
      • +

    Starting from scratch vs. re-using Components

    +

    It is always going to be more costly and riskier to develop something from scratch than to customise something that already exists.

    +

    You should make the software behind your projects open source. If many other organizations do the same, this allows code to be reused across jurisdictions. Not only does this ease the financial burden, but it helps create the expectation in populaces around the globe for the high quality engagement tools that their neighbouring country has access to.

    +

    That's not to say you should never develop something new - just ask around first, and make sure that what you are asking is technically feasible.

    +

    Commissioning New Software

    +

    Purchasing software is more closely related to having a piece of clothing made than to buying chairs. You can give the designer a basic vision of what you would like, but you will always need to come back to make sure it really fits, and your thoughts may change when you see things in practice. If you have an arrangement with your tailor which allows you to first specify the general idea, and a couple of other appointments for fittings and trials, you'll probably end up with a better and more creative result than if you tried to design the whole thing and it was simply unveiled to you at the end. You'll also feel more in control and it may even be quicker to do design and implementation in parallel.

    +

    Defining requirements & Designing the solution

    +

    Define the basic components of your project and prioritise them by their importance. As the developers start working on one of these chunks, you can then break it up into more specific tasks based on your evolved understanding of the project. A popular technique for this purpose are user stories, small narrative pieces that describe each problem: "As a [web site visitor] I want to [be able to see a supplier's contracts] so that I can [understand what services they provided to government]". The key to these stories is that they describe the actual user need, not the details of the solutions that you have envisaged. While you should of course discuss those with the developers as well, defining solutions is mainly the job of the developers, not the project manager.

    +

    Implementation: iterations and milestones

    +

    A saying amongst developers goes: "Walking on water and developing software from a specification are easy if both are frozen." As your software project is progressing you will likely realize that the specifications you have given need to be revised or extended. Yet by modifying the requirements you are essentially shifting the ground on which developers are executing - meaning they will have to stop their work to adjust. To prevent such changes from freezing all development, the process of introducing changes and additional requirements needs to be structured.

    +

    Iterations are periods of a defined length - often two or four weeks - during which developers are tasked to execute a set of previously selected user stories or requirements. Before the iteration starts, developers have to pull in the work from a list of tasks (a so-called backlog) prepared by the project manager, committing themselves to delivering those tasks within the agreed period. Crucially, project managers are not allowed to extend or revise the scope of an iteration while it is ongoing (unless they want to declare it failed). This method ensures that changes are introduced in bulk and understood by the team. This approach mandates the opposite of the more common unstructured communication between managers and developers, e.g. emails with unsorted lists of change requests which tend to be ignored and lead to confusion.

    +

    Whenever you consider an additional requirement, be sure to consider if it is realistic within the resources you have available. "Scope creep", the progressive extension of a project during its development, is a common cause of project failure. By becoming more and more ambitious, the project finally ends up with no usable product at all. To avoid scope creep, make sure to have a storage area for long-term ideas. Also make sure that developers accept additional tasks through a pull process, and not by having them pushed into their workflow.

    +

    Maintenance

    +

    Make sure to budget a for ongoing maintenance after the end of your project. Who is going to guarantee that the servers stay online? Who is going to fix a typo? It is unlikely that your project will remain entirely static after its initial development, so you should have an explicit agreement with the developers regarding future support. It is also useful to collect feedback after the projects release to commission a small number of additional days when enough additional work has accumulated.

    +

    Roles and how to find developers

    +

    The key ingredient to a successful software development project is having the right people on staff or as contractors. Depending on the scope and type of your project, you may need a variety of skills - these are some of the common descriptions:

    +
    • Web designers typically produce designs and layouts for web pages, often initially in a graphics program like Adobe Photoshop. Most, but not all, web designers then translate their designs into web markup (HTML, CSS).
      • +
    • Web developers are more technical. They produce interactive web interfaces such as search masks, browsers or specific form-based operations. They often use programming languages such as PHP, Ruby on Rails, Python or JavaScript.
      • +
    • Visualization designers develop graphics that represent quantitative information. A key distinction here is between non-interactive graphics (i.e. static images) and interactive visualizations, which often require some programming. There are still very few designers who design interactive visualizations, so rates may be relatively high.
      • +
    • Software developers are even more technical, developing backend software for data processing or acquisition. They are experienced in the use of database software (such as SQL databases) and programming languages such as Ruby, Python or Java.
      • +
    • Data scientists and statisticians produce analysis based on large sets of data, detecting tendencies and outliers in the dataset. They are not usually expected to produce front-end applications, but may produce software in the process of analysing data.
      • +
    • Usability experts and user experience (UX) designers think about the way your user will interact with your site answering questions such as 'is it obvious from the landing page what the purpose of this site is?'
      • +
    • Testers try and break things to test their robustness. This is particularly useful e.g. if you think your project will receive a lot of traffic as a result of a media campaign, you want to know your site can survive the hit
      • +

    Good places to look for developers

    +

    The easiest way to meet developers is through community meetups, such as hackdays. During such events, coders meet up to cooperatively develop prototypes of new software. To meet volunteer developers who can help you make sense and unleash the power of government spending and budgets, it's wortwhile to investigate events such as Random Hacks of Kindness (http://www.rhok.org), Data Kind (http://datakind.org/) and TechCamps (http://techcampglobal.org/).

    +

    There are a few ways you can discover if there is a hackday in your area. One way is to search on Lanyrd (http://lanyrd.com/search/?q=hackday&context=future) or set up an account on that system and request that you are alerted when there is a hackday in your area. Another approach is join mailing lists of organisations that might help you find developers e.g. the Open Knowledge Foundation lists (http://lists.okfn.org/mailman/listinfo) or the Sunlight Labs mailing list (https://groups.google.com/forum/#!forum/sunlightlabs).

    +

    What do things cost?

    +

    It's impossible to give concrete guidelines on how much a project should cost. Developers' salaries are generally quite high for a country's average, but vary very strongly from country to country. Worried about your project spiraling out of control? We'd recommend agreeing on a price per iteration, and it may be a good idea to draw up a contract which allows you to break it off if you are not happy with the work at the end of an iteration. Plus, you can generally also find a friendly developer to glance over a quote from a company for a sanity check.

    Presentation and engagement
    9. Presentation and engagement
    10. Selecting methods and tools

    9. Presentation and engagement

    +

    While raw financial data is valuable just as a tool for research, analysis and investigation, it can also be directly presented to an audience in a wide variety of ways. Such presentations can take the form of press releases in which data is used to make an argument, interactive infographics which condense information to visually represent a key finding or even searchable on-line databases that allow each user to find spending in their neighbourhood.

    +

    +

    Clarifying your topic

    +

    "It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away" - Antoine de Saint Exupéry

    +

    If you are reading this, you probably already have a fairly clear idea of the topic that you want to investigate and present. Still, it is useful to look at different ways of defining your subject. In some cases, you may merely want to give an overview of a particular area, e.g. by showing the proportions of government spending which go into the various functions of government. In other cases you may have a particular hypothesis that you want to portray or a particular journalistic story that you mean to tell, such as a case of systemic corruption that becomes evident only as you contrast one department's spending patterns with that of another one.

    +

    Sometimes there is a temptation to get lost in complexity. Using all the data and tools available to you to their full potential can force an almost endless amount of information onto our audiences. The benefit of doing so is limited, however: people will not understand the point that you are making, or, even worse, ignore you.

    +

    At the same time, technology can also be used to do the opposite: focus in on the key points, hiding other detail. Selecting where to go into depth doesn't have to be the author's responsibility any longer, it can be a choice that the user of data-driven applications is empowered to make based on their own interests. The prerequisite for using technology to create a clear focus is answering the question: What is the core of what we mean to present, what is the essence of the narrative our project is intended to convey?

    +

    Setting an objective

    +

    As you focus the topical core of your project, you also need to think about the objectives of your project: is it intended to inform the general public about an aspect of government budgeting or spending practices? Are you making a specific case to government, e.g. by proposing a new policy which you have gathered evidence and developed arguments for?

    +

    Your project can also serve as a forum for discussions, which would usually require you to facilitate the debate to maintain ongoing engagement.

    +

    The objective may be to establish a network-based, collaborative effort, e.g. to build a community that may review documents released by the government or evaluate the conditions of public contracts. Such modes of engagement require extensive design to encourage users to contribute in a way which best matches their particular skills and interests.

    +

    Some example of working with the public to collaboratively clean up financial data is Linked Gov (http://linkedgov.org/). This is a platform that will allow the public and civil servants to access and do small tasks that will clean government financial data.

    +

    Targeting an audience

    +

    The efficiency of projects based on spending data depends heavily on how well you have defined and understood your audience. If your message is too general for a specialized audience, it will be ignored. At the same time, a complicated and very technical presentation will usually fail to address broader parts of the population. Whether it be that your main audience is a specific constituency, government officials, the general public or a combination of those, it is important that the media and methods you apply help to get the message(s) across to the audience that you are targeting.

    +

    The general public

    +

    Is the purpose of your campaign to stimulate discussion amongst the general public? Unless your organization has exceedingly well-developed outreach channels, you will not usually be able to address the general public directly. Still, thanks to the social media and internet-based direct communication, many organizations can now contact and involve citizens who have shown some interest in their work directly. In cases where your main audience is the general public, simplicity is a must. It is necessary that the vast majority of people get a grasp of what you are trying to communicate. In these cases you may need to emphasize the visual presentation of the data that gets quickly and succinctly to the point, such as infographics. See the case study at the end of this chapter for how an organisation in Nigeria used Social Media to reach out to the general public.

    +

    Mass media

    +

    Unlike direct channels, broadcast and print media require you to first convince an intermediary of the relevance and quality of your message. Media are often the best channels to use when we want to create social pressure on governments to change some budget issue.

    +

    Community and constituency

    +

    This may include people and organizations with which you have previously cooperated or which are active in the same domain. Unlike the wider public, they are more likely to be interested not just in your conclusions, but also in the data and tools that you have used to reach your conclusions. Creating an open space for your community to discuss your work is essential in getting relevant feedback. When you need to reach to your constituency it may be necessary to provide a broader context, such as a full report including methodology, conclusions and possible paths of further discussion.

    +

    Government

    +

    Officials are often a target of your outreach, since they are in the position of actually deciding and implementing changes in policy. While they can be reached through public pressure and the media, your organization or some of its constituency may also choose to contact governments directly. In the case of trying to reach government officials, a balance of the previous approaches is very important. Meetings with government officials are rare and short-lived. Condense your main argument and supporting evidence into one page or less, and include a clear action that you want them to take.

    +

    Obviously, on any given issue you may choose to involve any set of these groups, but it is still important to have a clear picture of who they are and what their incentives motivate them.

    +

    +

    Case study

    +

    BudgIT Nigeria on social media

    +

    Data is invaluable. To the uninformed person where information asymmetry thrives, access to data illuminates the path to facts and provokes emotions that trigger results. Nevertheless, poor handling of data puts valuable facts in an opaque structure that communicates nothing. Data could be in a maze of thick document riddled with complex terms or iterations of figures that doesn’t connect the user. Not being able to promote discussion nor give provide contextual understanding, data may could be worthless.

    +
    +

    Nigeria returned to democracy in 1999 after lengthy years of military rule. Data under the barrel of the gun was an exclusive preserve of rulers as probing the facts behind data is taken as an affront to authority and strive to question the stained reputation of the junta. Civil servants were bound by the Official Secrets Act not to share government information hereby putting the citizens in the dark. Even after thirteen years of return to democracy, there is clearly a gap in accessing public data with some government officials still stricken with military era hangover. Data especially in terms of public expenditure communicates little to the larger sections of public who are not versed in financial accounting nor have requisite understanding to evaluate the complex arithmetic.

    +

    BudgIT, creative start-up sees a huge opportunity in using creative data visualization to stimulate interests concerning public expenditure. Understanding the ubiquity of the mobile device within the Nigerian locality and the increasing number of Nigerians online, BudgIT sees the opportunity to engage Nigerians and explain public expenditure in a simpler way. This thrives on building engagement across all platforms, encouraging the community to action via NGOs and reaching out to everyone to promote citizen participation. This is about making public data a social object and building an extensive network that demands change.

    +

    Building The Community

    +

    Across our interactions with users, we see a gulf in understanding what the budget is and what the citizens usually expect. We have engaged over 10,000 Nigerians over the budget and we profile them into three to ensure optimum value is delivered. The categories are briefly explained below:

    +

    Singular Users: These are users who want information in simple and quick format. There are interested in data but not in a deeper scale of analyzing the content or probing further. A series of simple tweets is enough for them to retell the stories or interactive applications that give a snapshot.

    +

    Closed Loop Users: A set of users who stimulate a discussion pattern, engage the data channel effectively to increase their knowledge of subject matter or challenge the assumptions of data. These set of users are adherents of the platform via social media, hence they ensure in building the platform through feedback systems or referral to their social connections.

    +

    The Data Hog: This set of users want the raw datasets to rework visualization or do extensive analysis for personal or enterprise purposes. We simply give them the data for their definitive purposes.

    +

    Engaging the Citizen

    +

    Across every society lies a literacy span and engaging every component is highly critical to the societal growth and stability. There is always a storyboard in every life that volumes of data can be matched with. People constantly want to be more informed especially concerning issues that they find difficult to understand. Engaging citizens is to take critical analysis of the target users at a time and itemize the possible profiles. A look at user profile demands a thorough analysis of their empathy, attention and insight towards the data available to them. What does the Nigerian citizen care about? Where is the information gap? How quickly can we reach out to them and place data in the storyboard of their lives? A critical understanding of the user’s psychology and the perceived response to the data is the first needed level of analysis. BudgIT’s immediate reach is to the average literate Nigerian connected to online forums and social media. Most online users amidst the array of interests in gaming, reading and sharing social connections within a limited timeframe will definitely need data in a brief and concise manner. After a snapshot of data either as a tweet or infographics, there’s an opportunity to build linked data on other platforms where the big picture can be set and interaction can be enhanced. <p >An important angle of visualization to us is understanding the data appreciation level of the users. Complex diagrams, superb infographics and aesthetic interactive applications might not convey the exact meaning to user based on his/her previous approach to data. Data vizualization needs to take into consideration how users can easily grasp the vizualized data and subject it to personal interpretation. A good visualization transfers knowledge and mostly important brings forth a story the user can easily connect with.

    +

    For us in BudgIT, our engagement model is anchored on the following:

    +
    • Stimulating discussion around Trends: In engaging with users on public data, BudgIT keeps track of online and offline discussions and seeks to provide data to enliven the interactions. A glaring example was the fuel strikes in January 2012 where there was a constant agitation among the protesters on the need to reduce the size of governance. BudgIT tracking the discussion via social media in 36 active hours quickly built an app that allows citizens to cut the Nigerian budget. The huge response with over 3,000 users who interacted with the budget using the app refined our engagement model. We keep looking for trends in the polity and matching it with relevant data quickly rendered into tweets or infographic display that quickly  extends our influence.
      • +
    • Constructive feedback mechanism and Balanced outlook: Data speaks volumes and individuals subject it to personal interpretations. In the engagement with users, feedback is enabled through discussion boards or retweets. Most users throw up discussions that tend to ask about stories behind the data and seeking opinions of BudgIT. Its of utmost priority to ensure that opinions only explain the facts behind the data and does not conform to individual disposition to the subject matter. It’s most necessary to build up feedback channels and engage the users creatively to ensure the community built around the data is sustained.
      • +
    • Localize Outlook of Data: For a dataset targeted at a particular group, BudgIT is building competency to localize its content and promote a channel of discussion that connects to the users. This involves taking a cultural outlook at the icons, symbols, objects and language to ensure that engagement concerning the budget flows seamlessly. This phase of engagement is with the grassroots who mostly don’t have access to Internet but only possess SMS-based mobile phones.
      • +

    After making the public expenditure data available in an easy-to-read format as shown on our portal (http://yourbudgit.com), we reach out to the citizens through the civil society organizations to ensure that citizens monitor capital projects in the budget. We also plan to develop a participatory framework where citizens and government institutions can meet in town halls to define key items in the budget that needs to be prioritized. Once we get citizens to be aware of capital projects in the budget and connect them with civil societies where BudgIT is not located, citizens can track report projects and report status. Ensuring that citizens of any literacy span are armed with data and possess a clear path to demand action, BudgIT is crossing the rubicon from open data to open action.

    10. Selecting methods and tools

    +

    The key to using data to tell people about your issue is to choose a really good story. The key to finding a good story is to formulate the right question for your audience.

    +

    Be guided by what interests you, as this stands a good chance of interesting other people. If you don't feel confident about your judgement then try lots of things that you consider interesting and see if it makes impact on the people you care about. Are they sharing your work on social media? Do they email you or link to your work from their own sites? If the answer is yes then you are really making good data stories.

    +

    Of course rigour and accountability are essential and you should always include links to your methodology and data, but it doesn't need to obscure your headline. Telling a story is the difference between people not reading your work and making it all worth the effort.

    +

    If you've already got data products that you want to share, try disseminating them in a couple of different ways. In addition to the methods outlined below, try preparing primers or manuals for other groups that want to start working with datasets that you may be particularly experienced with. You can also organize workshops, meetups, and webinars to familiarize groups with the data you work with. If grassroots issue groups can internalize your data products and present them in a contextually relevant way to their user base, then both groups have gained something from the partnership.

    +

    This is a collection of (hopefully) inspiring examples where groups have connected to the public directly through their websites, social media, or through the mainstream media. If you are interested in making media but don't have the capacity then fear not! Check out the first section of the book on reaching out to other organizations for help or running your own project.

    +

    Communicating via the web

    +

    If you are creating or adding to a website that you want citizens to access directly then here are some of the ways you can really use budgets or spending data to increase the value of your organisation.

    +

    Firstly we highly recommend that if you have some really great data, that's a story in itself, then don't waste any time - put the data up on your website and write a great story to go with it. Make sure the full dataset is available to download, ideally in a couple of formats like csv and json, and explain your methods. Then encourage your supporters to use and play with the data, add comments and talk in your community site or social networks.

    +

    If you feel your data needs that little bit more work to make it interesting or digestible, here are some examples of portals, interactive infographics, images, reports, and a searchable dataset.

    +

    Infographics and images

    +

    The art of the infographic is to make the complex look simple. This can be invaluable if you can communicate huge amounts of data in one picture. It is always recommended, both as a boost to your credibility and to allow others to extend your work, to publish the complete data needed to recreate the image. This allows your community of supporters to create their own visualisation and check your work easily before republishing it.

    +

    Below is an infographic summarizing the history of the US Federal Budget, by the Congressional Budget Office (CBO) /http://www.cbo.gov/publication/42636). It shows the budget over time and as a percentage of GDP, and in the context of total spending and revenue. This was used to emphasise that the federal government's budget deficit for 2011 was the third-largest in the past 40 years:

    +

    +

    Images of spending

    +

    This "binder full of burgers" infographic was created to show the sandwich of spending for Barack Obama's 2012 presidential election campaign. The data is not especially novel but the graphic is fun and engaging.

    +

    burger full binders

    +

    Interactive infographics

    +

    An interactive infographic can add even more data to a single screen full of images. A simple example is a bar chart where hovering over a bar with the cursor results in a dialogue box that tells more detail, like the exact value of that bar.

    +

    It seems there are two types of interactive graphics that work very well. One is where the user finds their own path through a complicated landscape of data. For example a map of the world where you can zoom in to details of each country or regions within a country and discover some facts about that area. The other is where the infographic guides the user through a story. For spending data in particular, people often like the ability to "drill down" in an infographic, or in other words, start with a high level view of the data and then gradually view more detail about different sections that they can select.

    +

    In recent years we have seen the quality of interactive infographics grow and become easier to produce. Anyone with a Google account can now make maps (http://www.google.com/drive/start/apps.html#fusiontables) and charts (https://developers.google.com/chart/). Equally, the tools to manipulate and display data in more sophisticated ways have become more accessible with efficient and usable javascript libraries. Using these libraries requires a bit of technical skill, so you may want to reach out to other organizations, as described in the first section of this book.

    +

    Budget interactives

    +

    An interactive graphic gives you the perfect opportunity to allow people to choose their own budget. This interactive chart called Citizen Budget is from the non-profit Open North. It shows Canadian budgets and gives citizens the chance to play at being the official controller. They can cut spending in any area they choose but the budget has to balance.

    +

    http://citizenbudget.com/

    +

    Spending interactives

    +

    The World Bank have a spending database called Boost. It's a technical collaboration between the World Bank and various governments to digitize their spending data. It is especially targeted at countries that wouldn't otherwise have the technical resources. One of these countries is Kenya and the data has been displayed to great effect here:

    +

    +
    The examples given here for interactive infographics showing both budget and spending data are examples of where the user can navigate for themselves. Please let us know if you have examples of step-by-step guides to Budgets and Spending as it would be great to include them here.
    +

    Online reports

    +

    A report allows you to expand on points and show lots of different views of a dataset that would be just too cluttered on one infographic or in an interactive. A report is a good place for a step-by-step guide through a more extensive study.

    +

    An example of a very short report is from Enough is Enough (EiE), a coalition of Nigerians aiming to enhance citizen engagement and good governance. Their EiE report on the state of the Nation shows a very concise summary of the major spending corruption stories they identified that year, along with some links to sources.

    +

    Podcasts and videos

    +

    A well made podcast will almost certainly be shared widely given the right push. Describing the significance of your data and getting expert comments is so much easier to digest in a podcast than a long report - even if they are both very well written. Again, the technology to make radio has radically reduced in price in recent years.

    +

    The NPR Planet Money podcast rose to fame for its incredibly accessible, award-winning coverage of the 2008 Global Financial Crisis. Planet money now covers financial issues all over the world on its weekly podcast.

    +

    Online portal

    +

    A portal is suitable for a large and relatively frequently updated dataset. People in your community can then build web applications to pull the data. An example of a portal that might be used to add more context to spending data is the police.uk website. This is a complete view, updated monthly, of crimes committed in the UK, with details down to street level. The site also provides an API[15] which you can use to add new data to an app showing, for example, spending on police with crime levels overlaid on map.

    +

    Mobile technologies

    +

    There are a few free and open tools that will allow you to display your data through a mobile phone or tablet app. AKVO is a non-profit foundation and they describe (http://www.akvo.org/blog/?p=4822) how their AKVO FLOW (http://www.akvo.org/blog/?p=4836) platform works using free open source software to communicate your data through an Android mobile app.

    +

    Social media

    +

    Building a community around your cause in general and data analysis in particular is really valuable. Setting up a mailing list and creating a space to collaborate like on a wiki can lead to some really interesting social connections at relatively low cost can allow you to work with some really devoted and knowledgeable people. Some nice examples are the OKFN mailing lists (http://lists.okfn.org/mailman/listinfo) using Mail Man - http://www.gnu.org/software/mailman/index.html), wikis (for example http://www.wikispaces.com/).

    +

    There are also the big social media companies like Twitter and Facebook of course! These are great for reaching large numbers of people who have anything from a passing interest to complete devotion to your cause. When you join create a Twitter account or Facebook page with the name of your organisation. The strategy when joining these is to contact some high profile tweeters or facebookers to take on your cause or message. Make sure there are plenty of links back to your website when you Tweet or post to Facebook or other platforms. If you have a data release then create a package of the data, the story and any other resources (image downloads, reports etc) on a webpage and then tweet a link out with a link to that page. The same principle applies with Facebook.

    +

    Communicating with the press

    +

    When writing a press release keep it simple and very relevant to a story the journalist might like to run - leaving descriptions of your organisation or how you got the data to the very end or ideally include only a link to a relevant webpage. To be relevant you need to hook into something in the news cycle (http://en.wikipedia.org/wiki/24-hour_news_cycle) or offer up such a strong story it would create news. If it is the latter, then you might want to give it to a number of newspapers with an embargo so they will release the story all at the same time to create a really big impact.

    +

    If you are looking for news hooks then at the simplest level the data can relate to the time of year, maybe when there is a new financial phase or an election and the news is focused very much on spending and budgets. For example if it's a complete cleaned up version of all public spending, or budget or all the pension funds, just showing this data qualifies as something a journalist would find interesting - they may even have an in-house team working on the same thing so will be pleased to have the data quicker. Here the press release can be quite simply describing the data if the data is detailed and relevant enough to stand alone, along with a link to your portal and some instructions for using it, or a download of the data.

    +

    If you're wondering about how much extra work you need to do to present your data before sending out a press release then the same principles apply as when adding content to your website. If you have a great story from a great piece of analysis or just great data alone, then send the data and the story in a concise form with links through to the methodology on your website.

    +

    One final tip before describing what you might include in your press release: when working with journalists be sure to get a link to your website included in the piece.

    +

    Including images

    +

    If you do have a dataset with a story that would really benefit from an infographic then add it to the press release. Or if you have a mobile or tablet app, a podcast or report then add a screenshot of the most enticing view of it too. Ideally attach a small copy of the image to the email and give a link to a higher resolution version on your website.

    +

    It's likely the newspaper's graphics team will create a version in the newspaper's style so you should include the data for your graphic in a usable format (no PDFs).

    +

    The journalist may also choose to use your graphic as it is, so be sure to add a small discrete copy of your logo in the corner of the image or other citation embedded in the image.

    +

    Including an interactive infographic

    +

    Equally you may have made a descriptive interactive, in which case you might want to make it 'embeddable' on other websites. You may also ask for the news site to link back your own website, or include a link in the embeddable portion of the infographic.

    +

    Communicating with governments

    +

    In modern democracies we would advise CSO's to influence government decision making with evidence-based, factual, easy to understand policy proposals. These proposals can be related to almost any particular subject, from energy efficiency, labour policy, immigration, minority issues to public finance. Government relation to these policy proposals can be decisive in times of elections, and for this reason they cannot ignore them. Government capacity is limited, so you might find it takes a while to get a response from them.

    +

    For CSOs it is important to differentiate between Government's general goals, and policy specific goals. For example, every responsible Government has a general goal of maintaining a balanced fiscal policy. As we get more specific about this goal, stakeholders' opinions begin to differ. Governments are rarely very clear about their policy goals, as this would make it easier for CSOs to keep the Government accountable. Real policy goals can also be very different from what Government declares as its policy goals. For example, every Government is against tax evasion, but in many cases Governments indirectly support them, one way or another.

    +

    Common themes for communicating

    +

    To summarize, the key themes are: have a great story, be clear, be connected to lots of other people and be accountable by publishing methodology and data. These apply equally if you are communicating with the public or the media.

    +

    Being clear is making easily repeated points with the data, or describing a juicy data set in one sentence. If you can make the presentation fun, fascinating, or just beautifully simple then you are on to a good thing.

    +

    Being connected is making things technologically very shareable and then facilitating the spread of your message through your own community of supporters and social media.

    +

    Being accountable is being completely transparent about how you obtained, cleaned and analysed the data. This method has to be explained very clearly too, and the standards maintained for every release until your CSO has a good reputation of trustworthiness. But please don't let concerns about trustworthiness stop your CSO from exploring and having fun with the data in the first place, which is really something you need to consider when settling on a data story.

    +

    The final points are wherever possible, have fun telling your story and don't be afraid to experiment with different styles and tones but don't delay if you have a great dataset on your hands - let the world know!

    Appendices
    11. Resources
    12. Glossary

    11. Resources

    +

    There are a variety of resources that you can use to accomplish some of the technical tasks described in the book. Some are geared towards users who are already experienced in a programming language, and some can be used by those with no experience. Most are helpful with analyzing and visualizing data.

    +

    For analysis and research

    +

    For visualisation and cleaning 

    +

    Free and Open Source:

    +

    Others

    +

    12. Glossary

    +

    aggregated

    +

    data that is summarized using statistical methods or by simply summing its component parts

    +

    API

    +

    Application Programming Interface: a specification allowing two pieces of software to interface with each other, without either having knowledge of the inner workings of the other

    +

    backlog

    +

    a list of tasks prepared by the project manager

    +

    bulk data

    +

    refers to the entirety of a dataset, instead of the incremental amounts that may be retrieval via other means, such as an API

    +

    CSO

    +

    Civil Service Organizations: a legally constituted organization created by natural or legal persons that operates independently from any form of government in the interest of civil society. Also commonly referred to as NGOs.

    +

    CSV

    +

    Comma Separated Values: A format for transactional data that separates each column value by a comma. Can be imported and exported from most spreadsheet programs

    +

    database query language

    +

    a syntax for interacting with data in a database. Usually specific to the software being used

    +

    hacker

    +

    one who programs enthusiastically (even obsessively) or who enjoys programming rather than just theorizing about programming (like a 'coder' - but more energetic :) )

    +

    hackday

    +

    a session where hackers come together to answer a particular task

    +

    iteration

    +

    a period of a defined length during which changes to a project can be made. Usually ends in a review meeting to plan the next iteration.

    +

    JSON 

    +

    Javascript Object Notation: a data format that is relatively easy for humans to read and write and very easy for machines to read and parse

    +

    Mirroring

    +

    creating an entire copy of a website on another server, primarily for backup

    +

    MySQL

    +

    (pronounced "My Sequel" or "MY S-Q-L"): Open source relational database software

    +

    nominatim

    +

    A web service which turns place names and addresses into coordinates so that they can be mapped / associated with an administrative area

    +

    PostgreSQL

    +

    (abbreviated Postgres): Open source object relational database software, a query language similar (but not identical) to that in MySQL is used

    +

    OCR

    +

    Optical Character Recognition: the electronic conversion of scanned images of handwritten, typewritten or printed text into machine-encoded text

    +

    reference data

    +

    data from outside an organisation (often from standards organisations) which is, apart from occasional revisions, static

    +

    scope creep

    +

    When a project's scope slowly and steadily increases, without a formal re-examination of the original terms and requirements

    +

    shapefile

    +

    a file format for geospatial vector data that is primarily used to feed data into mapping applications

    +

    Tilemill

    +

    Hosted and open source mapping software for creating maps

    +

    user story

    +

    short descriptions that outline in plain language the different use cases and possibilities for different types of users for the purposes of determining software requirements

    +

    vector data

    +

    a type of image composed of connected points rather than pixels

    +

    XML

    +

    eXtensible Markup Language: a markup language created to structure, store, and transport data by defining a set of rules for encoding documents in a format that is both human-readable and machine-readable.

    +

    Some of the above entries contain excerpts from Wikipedia.org and the Jargon File v. 4.4.8

    diff --git a/examples/openspending/content/resources/handbook/okfn-book.css b/examples/openspending/content/resources/handbook/okfn-book.css new file mode 100644 index 00000000..11dcc81f --- /dev/null +++ b/examples/openspending/content/resources/handbook/okfn-book.css @@ -0,0 +1,41 @@ + +@import url("http://assets.openspending.org/branding/fonts/opensans/stylesheet.css"); + +h1, h2, h3, h4, h5, +#booktitle, +.objavi-subsection-heading, +.objavi-chapter +{ + font-family: "OpenSansSemiBold"; +} + +.objavi-subsection-heading { + display: block; +} + +#frontmatterraw { + font-family: "OpenSansRegular"; +} + +#note { + padding: 10px; + padding-left: 15px; + padding-right: 15px; + border-radius: 5px; + background-color: #ddd; +} + + +span.section, +span.chapter, +span.pagenumber { + font-family: "OpenSansRegular"; +} + +p, li, #note, +.objavi-subsection, +#lightimage { + font-size: 15px; + line-height: 20px; + text-align: justify; +} diff --git a/examples/openspending/content/resources/handbook/spending-data-handbook.pdf b/examples/openspending/content/resources/handbook/spending-data-handbook.pdf new file mode 100644 index 00000000..ab8dab23 Binary files /dev/null and b/examples/openspending/content/resources/handbook/spending-data-handbook.pdf differ diff --git a/examples/openspending/content/resources/handbook/static/Boycott Starbucks.jpg b/examples/openspending/content/resources/handbook/static/Boycott Starbucks.jpg new file mode 100644 index 00000000..59cdc440 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/Boycott Starbucks.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-01 at 16.22.48.png b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-01 at 16.22.48.png new file mode 100644 index 00000000..acfbabbf Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-01 at 16.22.48.png differ diff --git a/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-01 at 16.22.48_1.png b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-01 at 16.22.48_1.png new file mode 100644 index 00000000..6f65963c Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-01 at 16.22.48_1.png differ diff --git a/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-13 at 5.30.32 PM.png b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-13 at 5.30.32 PM.png new file mode 100644 index 00000000..fcf0e8cf Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-13 at 5.30.32 PM.png differ diff --git a/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-15 at 11.34.49 AM.png b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-15 at 11.34.49 AM.png new file mode 100644 index 00000000..98ef826d Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-15 at 11.34.49 AM.png differ diff --git a/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-15 at 15.35.38.png b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-15 at 15.35.38.png new file mode 100644 index 00000000..a2f87848 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/Screen Shot 2012-11-15 at 15.35.38.png differ diff --git a/examples/openspending/content/resources/handbook/static/The-world-of-chocolate-ma-009.jpg b/examples/openspending/content/resources/handbook/static/The-world-of-chocolate-ma-009.jpg new file mode 100644 index 00000000..330c964b Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/The-world-of-chocolate-ma-009.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/boost-kenya.png b/examples/openspending/content/resources/handbook/static/boost-kenya.png new file mode 100644 index 00000000..9a42d958 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/boost-kenya.png differ diff --git a/examples/openspending/content/resources/handbook/static/boost-kenya2.png b/examples/openspending/content/resources/handbook/static/boost-kenya2.png new file mode 100644 index 00000000..e94bf5ba Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/boost-kenya2.png differ diff --git a/examples/openspending/content/resources/handbook/static/boost-kenya_1.png b/examples/openspending/content/resources/handbook/static/boost-kenya_1.png new file mode 100644 index 00000000..aeb4bff7 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/boost-kenya_1.png differ diff --git a/examples/openspending/content/resources/handbook/static/brazilian-transparency.png b/examples/openspending/content/resources/handbook/static/brazilian-transparency.png new file mode 100644 index 00000000..3e3fe42b Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/brazilian-transparency.png differ diff --git a/examples/openspending/content/resources/handbook/static/british columbia.png b/examples/openspending/content/resources/handbook/static/british columbia.png new file mode 100644 index 00000000..1e8e2ef6 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/british columbia.png differ diff --git a/examples/openspending/content/resources/handbook/static/british columbia_1.png b/examples/openspending/content/resources/handbook/static/british columbia_1.png new file mode 100644 index 00000000..1e8e2ef6 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/british columbia_1.png differ diff --git a/examples/openspending/content/resources/handbook/static/budget-spending-data-3.jpg b/examples/openspending/content/resources/handbook/static/budget-spending-data-3.jpg new file mode 100644 index 00000000..904c9b76 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/budget-spending-data-3.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/budgetinfographic.png b/examples/openspending/content/resources/handbook/static/budgetinfographic.png new file mode 100644 index 00000000..ccc4da5d Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/budgetinfographic.png differ diff --git a/examples/openspending/content/resources/handbook/static/budgetinfographic2.png b/examples/openspending/content/resources/handbook/static/budgetinfographic2.png new file mode 100644 index 00000000..ac698634 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/budgetinfographic2.png differ diff --git a/examples/openspending/content/resources/handbook/static/cclear-spend-sunlight-foundation.png b/examples/openspending/content/resources/handbook/static/cclear-spend-sunlight-foundation.png new file mode 100644 index 00000000..9516574d Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/cclear-spend-sunlight-foundation.png differ diff --git a/examples/openspending/content/resources/handbook/static/citizen-budget.png b/examples/openspending/content/resources/handbook/static/citizen-budget.png new file mode 100644 index 00000000..0ec41543 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/citizen-budget.png differ diff --git a/examples/openspending/content/resources/handbook/static/cso-3.jpg b/examples/openspending/content/resources/handbook/static/cso-3.jpg new file mode 100644 index 00000000..b76292bc Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/cso-3.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/data-gov-uk.png b/examples/openspending/content/resources/handbook/static/data-gov-uk.png new file mode 100644 index 00000000..2c2b24ed Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/data-gov-uk.png differ diff --git a/examples/openspending/content/resources/handbook/static/data-gov-uk_1.png b/examples/openspending/content/resources/handbook/static/data-gov-uk_1.png new file mode 100644 index 00000000..2c2b24ed Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/data-gov-uk_1.png differ diff --git a/examples/openspending/content/resources/handbook/static/funding.jpg b/examples/openspending/content/resources/handbook/static/funding.jpg new file mode 100644 index 00000000..d60c0e7a Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/funding.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/funding.pdf b/examples/openspending/content/resources/handbook/static/funding.pdf new file mode 100644 index 00000000..053aada1 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/funding.pdf differ diff --git a/examples/openspending/content/resources/handbook/static/funding_1.jpg b/examples/openspending/content/resources/handbook/static/funding_1.jpg new file mode 100644 index 00000000..d8049639 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/funding_1.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/indonesia-spend.png b/examples/openspending/content/resources/handbook/static/indonesia-spend.png new file mode 100644 index 00000000..f40cdd9b Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/indonesia-spend.png differ diff --git a/examples/openspending/content/resources/handbook/static/korean-budget.png b/examples/openspending/content/resources/handbook/static/korean-budget.png new file mode 100644 index 00000000..b4b2e426 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/korean-budget.png differ diff --git a/examples/openspending/content/resources/handbook/static/money_crop.png b/examples/openspending/content/resources/handbook/static/money_crop.png new file mode 100644 index 00000000..cda424d8 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/money_crop.png differ diff --git a/examples/openspending/content/resources/handbook/static/money_huge.png b/examples/openspending/content/resources/handbook/static/money_huge.png new file mode 100644 index 00000000..f8c8c42b Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/money_huge.png differ diff --git a/examples/openspending/content/resources/handbook/static/money_small.png b/examples/openspending/content/resources/handbook/static/money_small.png new file mode 100644 index 00000000..aba98bb4 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/money_small.png differ diff --git a/examples/openspending/content/resources/handbook/static/nigeria-budget.png b/examples/openspending/content/resources/handbook/static/nigeria-budget.png new file mode 100644 index 00000000..93a0ca5c Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/nigeria-budget.png differ diff --git a/examples/openspending/content/resources/handbook/static/nigeria-budget2.png b/examples/openspending/content/resources/handbook/static/nigeria-budget2.png new file mode 100644 index 00000000..f44e307d Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/nigeria-budget2.png differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline-analysing-data.jpg b/examples/openspending/content/resources/handbook/static/pipeline-analysing-data.jpg new file mode 100644 index 00000000..ce5058bc Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline-analysing-data.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline-ask-a-question.jpg b/examples/openspending/content/resources/handbook/static/pipeline-ask-a-question.jpg new file mode 100644 index 00000000..ae290e80 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline-ask-a-question.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline-cleaning-data.jpg b/examples/openspending/content/resources/handbook/static/pipeline-cleaning-data.jpg new file mode 100644 index 00000000..a5ba9574 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline-cleaning-data.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline-getting-data.jpg b/examples/openspending/content/resources/handbook/static/pipeline-getting-data.jpg new file mode 100644 index 00000000..171aec8b Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline-getting-data.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline-presenting-data.jpg b/examples/openspending/content/resources/handbook/static/pipeline-presenting-data.jpg new file mode 100644 index 00000000..55004b16 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline-presenting-data.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline.jpg b/examples/openspending/content/resources/handbook/static/pipeline.jpg new file mode 100644 index 00000000..ed2e0ce5 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline.pdf b/examples/openspending/content/resources/handbook/static/pipeline.pdf new file mode 100644 index 00000000..82096895 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline.pdf differ diff --git a/examples/openspending/content/resources/handbook/static/pipeline_1.jpg b/examples/openspending/content/resources/handbook/static/pipeline_1.jpg new file mode 100644 index 00000000..2814e534 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pipeline_1.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pyramid.jpg b/examples/openspending/content/resources/handbook/static/pyramid.jpg new file mode 100644 index 00000000..236855c8 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pyramid.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/pyramid_1.jpg b/examples/openspending/content/resources/handbook/static/pyramid_1.jpg new file mode 100644 index 00000000..bcb1934d Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/pyramid_1.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/revenue-watch.png b/examples/openspending/content/resources/handbook/static/revenue-watch.png new file mode 100644 index 00000000..da3f0759 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/revenue-watch.png differ diff --git a/examples/openspending/content/resources/handbook/static/riot-release-athens-protest.n.jpg b/examples/openspending/content/resources/handbook/static/riot-release-athens-protest.n.jpg new file mode 100644 index 00000000..fcd09d9f Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/riot-release-athens-protest.n.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/sera-lione.png b/examples/openspending/content/resources/handbook/static/sera-lione.png new file mode 100644 index 00000000..213e3c2d Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/sera-lione.png differ diff --git a/examples/openspending/content/resources/handbook/static/tumblr_md0vl73RzF1rkwn28o1_1280.jpg b/examples/openspending/content/resources/handbook/static/tumblr_md0vl73RzF1rkwn28o1_1280.jpg new file mode 100644 index 00000000..8cbd1c93 Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/tumblr_md0vl73RzF1rkwn28o1_1280.jpg differ diff --git a/examples/openspending/content/resources/handbook/static/warning.png b/examples/openspending/content/resources/handbook/static/warning.png new file mode 100644 index 00000000..6413b54a Binary files /dev/null and b/examples/openspending/content/resources/handbook/static/warning.png differ diff --git a/examples/openspending/content/resources/handbook/update.sh b/examples/openspending/content/resources/handbook/update.sh new file mode 100755 index 00000000..bb668c84 --- /dev/null +++ b/examples/openspending/content/resources/handbook/update.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +curl -o book.zip http://okfn.booktype.pro/export/spending-data-handbook/export +unzip -o -d book book.zip + +python web_version.py + +mkdir -p static +rm static/* +cp book/static/* static/ + +mogrify -geometry '600x600>' static/* + + diff --git a/examples/openspending/content/resources/handbook/vendor/bookjs/book.css b/examples/openspending/content/resources/handbook/vendor/bookjs/book.css new file mode 100644 index 00000000..19154ccf --- /dev/null +++ b/examples/openspending/content/resources/handbook/vendor/bookjs/book.css @@ -0,0 +1,247 @@ +/* background color is grey */ + +body { +background-color:#efefef; +} + + +/* size of pages: A5 */ + +@page { +size:5.8in 8.3in; /* Setting seems to be ignored due to webkit bug */ +} + +.page { +height:8.3in; +width:5.8in; +page-break-after:always; +} + +/* height of pages when no CSS regions present: auto. Also add some padding. */ + +.page.simple { +height:auto; +padding: 1in; +} + +/* Every .page contains various elements (.contents, .header, .pagenumber) that together make out the page. The elements are placed with absolute coordinates in relation to the page and are, if not specified otherwise, centered. Pages have white background color */ + +.page { +position:relative; +margin-left:auto; +margin-right:auto; +background-color:#fff; +} + +/* When showing the page on the screen, put a border around the page and add a little margin beween the pages. */ + +@media screen{ +.page { +border:solid 1px #000; +margin-bottom:.2in; +} +} + +/* The .contentsContianer contains the .contents and the .footnotes */ + +.contentsContainer { +height:6.67in; +width:4.03in; +position:absolute; +bottom:.8in; +} + + + +/* images should not overflow the contents box */ + +img { +max-height: 6.57in; /* page height - .1in */ +max-width: 3.93in; /* page width - .1in */ +-webkit-region-break-before: always; +-webkit-region-break-after: always; +} + +.pagenumber { +margin-top:.2in; +position:absolute; +bottom:.4in; +left:0; +right:0; +font-size:15px; +} + +.header { +margin-bottom:.2in; +position:absolute; +top:.4in; +left:0; +right:0; +font-size:13px; +} + +/* Don't display section titles in the header on odd pages */ +.page:nth-child(odd) .header .section { +display:none; +} + +/* Don't display chapter titles in the header on even pages */ +.page:nth-child(even) .header .chapter { +display:none; +} + +/* Centering the page number on first pages of chapters */ +.chapter .page:first-child .pagenumber { +text-align: center; +margin-left: auto; +margin-right: auto; +} + + +/* ELEMENTS OF THE TABLE OF CONTENTS */ + +/* The title text of the TOC */ +#toc-title:before { +content:"Contents"; +} + +/* Styling of the TOC title */ +#toc-title { +font-size:22px; +font-weight:700; +text-align:left; +padding-bottom:.4in; +} + +/* Styling of individual toc entry */ +.toc-entry { +width:4.2in; +display:block; +padding-bottom:.3in; +} + +/* Page numbers of toc entries should float to the right and have normal font weight */ +.toc-entry .toc-pagenumber { +font-weight:400; +display:inline-block; +vertical-align:text-bottom; +font-size:16px; +float:right; +} + +/* special styling for TOC entries that represent sections */ +.toc-entry.section { +font-weight:700; +font-size:16px; +text-transform:uppercase; +padding-bottom:.3in; +} + +/* Left and right page gutter */ +.page:nth-child(odd) .contentsContainer { +right:.6in; +} + +.page:nth-child(even) .contentsContainer { +left:.6in; +} + +/* Left and right page page numbers and headers */ + +.page:nth-child(odd) .pagenumber,.page:nth-child(odd) .header { +margin-right:.6in; +text-align:right; +} + +.page:nth-child(even) .pagenumber,.page:nth-child(even) .header { +margin-left:.6in; +text-align:left; +} + +/* No page number on section start pages and on title page */ +.section .page:first-child .pagenumber,#frontmatter .page:first-child .pagenumber { +display:none; +} + +/* No header on section and chapter start pages */ +.section .page:first-child .header,.chapter .page:first-child .header { +display:none; +} + +/* Manual page breaks */ +.pagebreak { +-webkit-region-break-after:always; +} + +/* Work-around to make section title pages no longer than one page */ + +.objavi-subsection-heading { +line-height:120px!important; +font-size:22px; +font-weight:700; +text-align:left; +display:none; +} + +/* Custom CSS for elements that do not come directly out of body.html */ + +#booktitle { +margin-top:1.7in; +font-size:26px; +font-weight:700; +text-align:center; +} + +#booksubtitle { +font-size:18px; +margin-top:.2in; +font-weight:700; +text-align:center; +} + +#bookeditors { +padding-top:2in; +font-weight:700; +text-align:center; +font-size:12px; +} + +/* General CSS for contents */ + +p { +font-size:15px; +line-height:20px; +text-align:justify; +} + +h1 { +font-size:21px; +} + +h2 { +font-size:19px; +} + +h3 { +font-size:17px; +} + +/* Styling of footnotes */ + +.footnotes { +font-size:12px; +line-height: 1em; +margin:.25em; +} + +.footnote-reference { +font-size:10px; +font-weight: bold; +} + +/* If footnotes are present, a margin of .2in is added between .contents and .footnotes and a black line is drawn above the footnotes. */ + +.footnotes div:first-child { +padding-top: .5em; +border-top: 1px solid black; +} diff --git a/examples/openspending/content/resources/handbook/vendor/bookjs/book.js b/examples/openspending/content/resources/handbook/vendor/bookjs/book.js new file mode 100644 index 00000000..eb1601dc --- /dev/null +++ b/examples/openspending/content/resources/handbook/vendor/bookjs/book.js @@ -0,0 +1,696 @@ +/*! + * BookJS v.0.23.1-dev + * Copyright 2012 Aleksandar Erkalovic, Marita Fraser, Steven Levithan, + * Philip Schatz and Johannes Wilm. Freely available under the AGPL. For + * further details see LICENSE.txt + * + * Using this library you can turn an HTML element into a series + * of pages using CSS Regions. If the browser doesn't support CSS Regions, + * everything will be flown into one large page container that looks like a + * very long page. + * + * + * HOWTO + * + * In order to use this library, link to the corresponding file as well as this + * javascript file within your html code. If you need to set custom options, + * set them before including this file by defining an object named + * paginationConfig and setting the customization options as keys within this + * object. Like this: + * + * + * + * + * + * The following options are available to customize the pagination behavior. In + * the descriptions below you can see the default values for these options. You + * only need to specify the options if you want to deviate from the default + * value. + * + * sectionStartMarker: h1 -- This is the HTML element we look for to find where + * a new section starts. + * + * sectionTitleMarker: h1 -- Within the newly found section, we look for the + * first instance of this element to determine the title of the section. + * + * chapterStartMarker: h2 -- This is the HTML element we look for to find where + * a new chapter starts. + * + * chapterTitleMarker: h2 -- Within the newly found chapter, we look for the + * first instance of this element to determine the title of the chapter. + * + * flowElement: document.body -- This specifies element the container element + * of the content we will flow into pages. You can use any javascript selector + * here, such as "document.getElementById('contents')" . + * + * alwaysEven: false -- This determines whether each section and chapter should + * have an even number of pages (2, 4, 6, 8, ...). + * + * columns: 1 -- This specifies the number of number of columns used for the + * body text. + * + * enableFrontmatter: true -- This resolves whether a table of contents, page\ + * headers and other frontmatter contents should be added upon page creation. + * + * bulkPagesToAdd: 50 -- This is the initial number of pages of each flowable + * part (section, chapter). After this number is added, adjustments are made by + * adding another bulk of pages or deletin pages individually. It takes much + * less time to delete pages than to add them individually, so it is a point to + * overshoot the target value. For larger chapters add many pages at a time so + * there is less time spent reflowing text. + * + * pagesToAddIncrementRatio: 1.4 -- This is the ratio of how the bulk of pages + * incremented. If the initial bulkPagestoAdd is 50 and those initial 50 pages + * were not enough space to fit the contents of that chapter, then next + * 1.4 * 50 = 70 are pages, for a total of 50+70 = 120 pages, etc. . 1.4 seems + * to be the fastest in most situations. + * + * frontmatterContents: none -- These are the HTML contents that are added to + * the frontmatter before the table of contents. This would usually be a title + * page and a copyright page, including page breaks. + * + * autoStart: true -- This controls whether pagination should be executed + * automatically upon page load. If it is set to false, pagination has to be + * initiated manually by calling Pagination.applyBookLayout() or + * Pagination.applySimpleBookLayout() in case CSS Regions are not present. + * Check Pagination._cssRegionCheck() to see if CSS Regions are present. + */ +/* + * The Pagination object represents all the pagination functionality which is + * added to its namespace. + */ +var Pagination = new Object; + +Pagination.config = { + 'sectionStartMarker': 'h1', + 'sectionTitleMarker': 'h1', + 'chapterStartMarker': 'h2', + 'chapterTitleMarker': 'h2', + 'flowElement': 'document.body', + 'alwaysEven': false, + 'columns': 1, + 'enableFrontmatter': true, + 'bulkPagesToAdd': 50, + 'pagesToAddIncreementRatio': 1.4, + 'frontmatterContents': '', + 'autoStart': true +}; + +Pagination.initiate = function () { + this.userConfigImport(); + this.setStyle(); +} + +Pagination.userConfigImport = function () { + if (window.paginationConfig) { + for (var key in paginationConfig) { + Pagination.config[key] = paginationConfig[key]; + } + } +} + +Pagination.setStyle = function () { + var stylesheet = document.createElement('style'); + stylesheet.innerHTML = ".contentsContainer {display: -webkit-box; -webkit-box-orient: vertical;}" + " .contents {display: -webkit-box; -webkit-box-flex: 1}" + " .contents-column {-webkit-box-flex: 1}" + " .footnotes .invisible {visibility: hidden}" + " #contents .footnote {display: none}"; + document.head.appendChild(stylesheet); +} + +Pagination.romanize = function () { + // Create roman numeral representations of numbers. + var digits = String(+this.value).split(""), + key = ["", "C", "CC", "CCC", "CD", "D", "DC", "DCC", "DCCC", "CM", "", "X", "XX", "XXX", "XL", "L", "LX", "LXX", "LXXX", "XC", "", "I", "II", "III", "IV", "V", "VI", "VII", "VIII", "IX"], + roman = "", + i = 3; + while (i--) { + roman = (key[+digits.pop() + (i * 10)] || "") + roman; + } + return Array(+digits.join("") + 1).join("M") + roman; +}; + +Pagination.pageCounterCreator = function (selector, show) { + this.selector = selector; + if (show !== undefined) { + this.show = show; + } +}; + +Pagination.pageCounterCreator.prototype.value = 0; + +Pagination.pageCounterCreator.prototype.needsUpdate = false; + +Pagination.pageCounterCreator.prototype.show = function () { + return this.value; +}; + +Pagination.pageCounterCreator.prototype.incrementAndShow = function () { + this.value++; + return this.show(); +}; + + +Pagination.pageCounterCreator.prototype.numberPages = function () { + if (this.needsUpdate) { + this.value = 0; + this.needsUpdate = false; + + var pagenumbersToNumber = document.querySelectorAll('.page .pagenumber.' + this.selector); + for (var i = 0; i < pagenumbersToNumber.length; i++) { + pagenumbersToNumber[i].innerHTML = this.incrementAndShow(); + } + } +}; + +Pagination.pageCounters = {}; + +Pagination.pageCounters.arab = new Pagination.pageCounterCreator('arabic'); +Pagination.pageCounters.roman = new Pagination.pageCounterCreator('roman', Pagination.romanize); + +Pagination.createPages = function (num, flowName, pageCounterSelector, columns) { + // Create the DOM structure of each page. + var page, contents, footnotes, contentsContainer, column, topFloats; + var tempRoot = document.createDocumentFragment(); + for (var i = 0; i < num; i++) { + page = document.createElement('div'); + page.classList.add('page'); + + header = document.createElement('div'); + header.classList.add('header'); + + chapterheader = document.createElement('span'); + chapterheader.classList.add('chapter'); + header.appendChild(chapterheader); + + sectionheader = document.createElement('span'); + sectionheader.classList.add('section'); + header.appendChild(sectionheader); + + page.appendChild(header); + + pagenumberfield = document.createElement('div'); + pagenumberfield.classList.add('pagenumber'); + pagenumberfield.classList.add(pageCounterSelector); + + page.appendChild(pagenumberfield); + + // If flowName is given, create a page with content flow. + if (flowName) { + contentsContainer = document.createElement('div'); + contentsContainer.classList.add('contentsContainer'); + + topFloats = document.createElement('div'); + topFloats.classList.add('topFloats'); + + contents = document.createElement('div'); + contents.classList.add('contents'); + + for (var j = 0; j < columns; j++) { + column = document.createElement('div'); + column.classList.add('contents-column'); + contents.appendChild(column); + } + + footnotes = document.createElement('div'); + footnotes.classList.add('footnotes'); + + contentsContainer.appendChild(topFloats); + contentsContainer.appendChild(contents); + contentsContainer.appendChild(footnotes); + page.appendChild(contentsContainer); + // If no flowName is given, an empty page is created. + } else { + page.classList.add('empty'); + } + + tempRoot.appendChild(page); + } + return tempRoot; +}; + +Pagination.events = {}; + +Pagination.events.bodyLayoutUpdated = document.createEvent('Event'); + +Pagination.events.bodyLayoutUpdated.initEvent('bodyLayoutUpdated', true, true); + +Pagination.events.layoutFlowFinished = document.createEvent('Event'); + +Pagination.events.layoutFlowFinished.initEvent('layoutFlowFinished', true, true); + +Pagination.events.pageLayoutUpdate = document.createEvent('Event'); + +Pagination.events.pageLayoutUpdate.initEvent('pageLayoutUpdated', true, true); + +Pagination.events.footnotesNeedMove = document.createEvent('Event'); + +Pagination.events.footnotesNeedMove.initEvent('footnotesNeedMove', true, true); + +Pagination.headersAndToc = function (bodyObjects) { + + var currentChapterTitle = ''; + var currentSectionTitle = ''; + + var tocDiv = document.createElement('div'); + tocDiv.id = 'toc'; + + tocTitleDiv = document.createElement('div'); + tocTitleDiv.id = 'toc-title'; + + tocDiv.appendChild(tocTitleDiv); + + + + for (var i = 0; i < bodyObjects.length; i++) { + bodyObjects[i].findTitle(); + bodyObjects[i].findStartpageNumber(); + + if (bodyObjects[i].type == 'chapter') { + currentChapterTitle = bodyObjects[i].title; + } else if (bodyObjects[i].type == 'section') { + currentSectionTitle = bodyObjects[i].title; + } + var pages = bodyObjects[i].div.childNodes; + + for (var j = 0; j < pages.length; j++) { + var chapterHeader = pages[j].querySelector('.header .chapter'); + chapterHeader.innerHTML = currentChapterTitle; + + var sectionHeader = pages[j].querySelector('.header .section'); + sectionHeader.innerHTML = currentSectionTitle; + } + + if (bodyObjects[i].type) { + + var tocItemDiv = document.createElement('div'); + tocItemDiv.classList.add('toc-entry'); + tocItemDiv.classList.add(bodyObjects[i].type); + + var tocItemTextSpan = document.createElement('span'); + tocItemTextSpan.classList.add('toc-text'); + + tocItemTextSpan.innerHTML = bodyObjects[i].title; + tocItemDiv.appendChild(tocItemTextSpan); + + var tocItemPnSpan = document.createElement('span'); + tocItemPnSpan.classList.add('toc-pagenumber'); + + if (typeof bodyObjects[i].startpageNumber !== 'undefined') { + var tocItemPnText = document.createTextNode(bodyObjects[i].startpageNumber); + tocItemPnSpan.appendChild(tocItemPnText); + } + + tocItemDiv.appendChild(tocItemPnSpan); + + tocDiv.appendChild(tocItemDiv); + + } + + } + + return tocDiv; +}; + +/** + * Creates objects for each item in the body (section start, chapter) + */ + +Pagination.createBodyObjects = function () { + // + var bodyObjects = []; + var chapterCounter = 0; + + bodyObjects.push(new Pagination.flowObject('bodypre', Pagination.pageCounters.arab)); + + var bodyContainer = eval(Pagination.config.flowElement); + var bodyContents = bodyContainer.childNodes; + + + for (var i = bodyContents.length; i > 0; i--) { + + if (bodyContents[0].nodeType == 1) { + if (bodyContents[0].webkitMatchesSelector(Pagination.config.chapterStartMarker)) { + bodyObjects.push(new Pagination.flowObject('body' + chapterCounter++, Pagination.pageCounters.arab)); + bodyObjects[chapterCounter].setType('chapter'); + + } else if (bodyContents[0].webkitMatchesSelector(Pagination.config.sectionStartMarker)) { + bodyObjects.push(new Pagination.flowObject('body' + chapterCounter++, Pagination.pageCounters.arab)); + bodyObjects[chapterCounter].setType('section'); + } + } + + bodyObjects[chapterCounter].rawdiv.appendChild(bodyContents[0]); + } + + return bodyObjects; + +}; + +Pagination.applyBookLayout = function () { + + var bodyObjects = Pagination.createBodyObjects(); + + //Create div for layout + var layoutDiv = document.createElement('div'); + layoutDiv.id = 'layout'; + document.body.appendChild(layoutDiv); + + //Create div for contents + var contentsDiv = document.createElement('div'); + contentsDiv.id = 'contents'; + document.body.appendChild(contentsDiv); + + counter = 0; + + for (var i = 0; i < bodyObjects.length; i++) { + layoutDiv.appendChild(bodyObjects[i].div); + contentsDiv.appendChild(bodyObjects[i].rawdiv); + bodyObjects[i].initiate(); + } + + Pagination.pageCounters.arab.numberPages(); + + if (Pagination.config.enableFrontmatter) { + //Create and flow frontmatter + fmObject = new Pagination.flowObject('frontmatter', Pagination.pageCounters.roman, 1); + fmObject.columns = 1; + contentsDiv.insertBefore(fmObject.rawdiv, contentsDiv.firstChild); + fmObject.rawdiv.innerHTML = Pagination.config.frontmatterContents; + var toc = Pagination.headersAndToc(bodyObjects); + fmObject.rawdiv.appendChild(toc); + layoutDiv.insertBefore(fmObject.div, bodyObjects[0].div); + fmObject.initiate(); + var redoToc = function () { + var oldToc = toc; + toc = Pagination.headersAndToc(bodyObjects); + fmObject.rawdiv.replaceChild(toc, oldToc); + }; + document.body.addEventListener('bodyLayoutUpdated', redoToc); + } + document.dispatchEvent(Pagination.events.layoutFlowFinished); +}; + + + +Pagination.applySimpleBookLayout = function () { + // Apply this alternative layout in case CSS Regions are not present + bodyContainer = eval(Pagination.config.flowElement); + simplePage = document.createElement('div'); + simplePage.classList.add('page'); + simplePage.classList.add('simple'); + simplePage.innerHTML = bodyContainer.innerHTML; + simplePage.id = bodyContainer.id; + bodyContainer.innerHTML = ''; + document.body.appendChild(simplePage); +}; + +Pagination._cssRegionsCheck = function () { + // Check whether CSS Regions are present in Chrome 23+ version + if ((document.webkitGetNamedFlows) && (document.webkitGetNamedFlows() !== null)) { + return true; + } + return false; +}; + +Pagination.autoStartInitiator = function () { + // To be executed upon document loading. + var cssRegionsPresent = Pagination._cssRegionsCheck(); + if ((document.readyState == 'interactive') && (!(cssRegionsPresent))) { + Pagination.applySimpleBookLayout(); + } else if ((document.readyState == 'complete') && (cssRegionsPresent)) { + Pagination.applyBookLayout(); + } +} + + + +Pagination.flowObject = function (name, pageCounter) { + this.name = name; + this.pageCounter = pageCounter; + + this.rawdiv = document.createElement('div'); + this.rawdiv.id = name + 'raw'; + + this.div = document.createElement('div'); + this.div.id = name; + + this.bulkPagesToAdd = Pagination.config.bulkPagesToAdd; + + this.columns = Pagination.config.columns; + + this.footnotes = []; +}; + +Pagination.flowObject.prototype.totalPages = 0; + +Pagination.flowObject.prototype.redoPages = false; + +Pagination.flowObject.prototype.overset = false; + +Pagination.flowObject.prototype.firstEmptyRegionIndex = -1; + +Pagination.flowObject.prototype.initiate = function () { + this.setStyle(); + this.namedFlow = document.webkitGetNamedFlows()[this.name]; + this.addOrRemovePages(); + this.setupReflow(); + this.findAllFootnotes(); + this.addFootnoteReferences(); + this.layoutFootnotes(); + this.setupFootnoteReflow(); + this.pageCounter.numberPages(); +} + +Pagination.flowObject.prototype.setStyle = function () { + var stylesheet = document.createElement('style'); + stylesheet.innerHTML = "#" + this.name + " .contents-column {-webkit-flow-from:" + this.name + ";}" + " #" + this.name + "raw {-webkit-flow-into:" + this.name + ";}"; + document.head.appendChild(stylesheet); +} + +Pagination.flowObject.prototype.setType = function (type) { + this.type = type; + this.div.classList.add(type); +}; + +Pagination.flowObject.prototype.findTitle = function () { + var titleField; + if (this.type == 'chapter') { + titleField = this.rawdiv.querySelector(Pagination.config.chapterTitleMarker); + this.title = titleField.innerHTML; + } else if (this.type == 'section') { + titleField = this.rawdiv.querySelector(Pagination.config.sectionTitleMarker); + this.title = titleField.innerHTML; + } +}; + +Pagination.flowObject.prototype.findStartpageNumber = function () { + if (this.rawdiv.innerText.length > 0) { + var startpageNumberField = this.div.querySelector('.pagenumber'); + this.startpageNumber = startpageNumberField.innerText; + } +}; + +Pagination.flowObject.prototype.findFootnoteReferencePage = function (footnoteReference) { + return this.namedFlow.getRegionsByContent(footnoteReference)[0].parentNode.parentNode.parentNode; +} + +Pagination.flowObject.prototype.findFootnotePage = function (footnote) { + return footnote.parentNode.parentNode.parentNode; +} + +Pagination.flowObject.prototype.compareReferenceAndFootnotePage = function (footnoteObject) { + // Check whether a footnote and it's corresponding reference in the text are on the same page. + var referencePage = this.findFootnoteReferencePage(footnoteObject['reference']); + var footnotePage = this.findFootnotePage(footnoteObject['item']); + + if (footnotePage === referencePage) { + return true; + } else { + return false; + } +} + + +Pagination.flowObject.prototype.setupFootnoteReflow = function () { + var flowObject = this; + + var checkAllFootnotePlacements = function () { + flowObject.checkAllFootnotePlacements(); + } + + this.namedFlow.addEventListener('webkitRegionLayoutUpdate', checkAllFootnotePlacements); + + var reFlow = function () { + flowObject.namedFlow.removeEventListener('webkitRegionLayoutUpdate', checkAllFootnotePlacements); + + flowObject.layoutFootnotes(); + + flowObject.namedFlow.addEventListener('webkitRegionLayoutUpdate', checkAllFootnotePlacements); + } + + this.namedFlow.addEventListener('footnotesNeedMove', reFlow); +} + +Pagination.flowObject.prototype.checkAllFootnotePlacements = function () { + for (var i = 0; i < this.footnotes.length; i++) { + if (!(this.compareReferenceAndFootnotePage(this.footnotes[i]))) { + this.namedFlow.dispatchEvent(Pagination.events.footnotesNeedMove); + } + } +} + +Pagination.flowObject.prototype.findAllFootnotes = function () { + var allFootnotes = this.rawdiv.getElementsByClassName('footnote'); // Look for all the items that have "footnote" in their class list. These will be treated as footnote texts. + for (var i = 0; i < allFootnotes.length; i++) { + var footnoteObject = {}; // We create this object so that we can find the footnote item and reference again later on. + + footnoteObject['original'] = allFootnotes[i]; + + var numFootnote = document.createElement('sup'); // Create a sup-element with the class "footnote-reference" that holds the current footnote number. This will be used both in the body text and in the footnote itself. + numFootnote.classList.add('footnote-reference'); + numFootnoteContents = document.createTextNode(i + 1); + numFootnote.appendChild(numFootnoteContents); + + var footnote = document.createElement('div'); // Put the footnote number and footnote text together in a div-element with the class footnote-item + footnote.classList.add('footnote-item'); + footnote.classList.add('visible'); + footnote.appendChild(numFootnote); + + var footnoteText = allFootnotes[i].cloneNode(true); + footnote.appendChild(footnoteText); + + footnoteObject['item'] = footnote; + + numFootnoteReference = numFootnote.cloneNode(true); + + footnoteObject['reference'] = numFootnoteReference; + + this.footnotes.push(footnoteObject); + } +} + +Pagination.flowObject.prototype.addFootnoteReferences = function () { + for (var i = 0; i < this.footnotes.length; i++) { + this.footnotes[i]['original'].parentNode.insertBefore(this.footnotes[i]['reference'], this.footnotes[i]['original']); // Insert the footnote number in the body text just before the original footnote text appears). + } +} + +Pagination.flowObject.prototype.layoutFootnotes = function () { + for (var i = 0; i < this.footnotes.length; i++) { + + if ('hidden' in this.footnotes[i]) { + this.footnotes[i]['hidden'].parentNode.removeChild(this.footnotes[i]['hidden']); + delete this.footnotes[i]['hidden']; + } + + var footnoteReferencePage = this.findFootnoteReferencePage(this.footnotes[i]['reference']); // We find the page where the footnote is referenced from. + var currentFootnoteContainer = footnoteReferencePage.querySelector('.footnotes'); + currentFootnoteContainer.appendChild(this.footnotes[i]['item']); // We insert the footnote in the footnote container of that page. + + if (!(this.compareReferenceAndFootnotePage(this.footnotes[i]))) { + // If the footnote reference has been moved from one page to another through the insertion procedure, we set the visibility of the footnote to "hidden" so that it continues to take up the same space and then insert it one more time on the page from where it now is referenced. + this.footnotes[i]['hidden'] = this.footnotes[i]['item']; + this.footnotes[i]['item'] = this.footnotes[i]['hidden'].cloneNode(true); + + this.footnotes[i]['hidden'].classList.remove('visible'); + this.footnotes[i]['hidden'].classList.add('invisible'); + + footnoteReferencePage = this.findFootnoteReferencePage(this.footnotes[i]['reference']); // We find the page where the footnote is referenced from now. + currentFootnoteContainer = footnoteReferencePage.querySelector('.footnotes'); + currentFootnoteContainer.appendChild(this.footnotes[i]['item']); + } + } +}; + +Pagination.flowObject.prototype.makeEvenPages = function () { + var emptyPage = this.div.querySelector('.page.empty'); + if (emptyPage) { + this.div.removeChild(emptyPage); + } + var allPages = this.div.querySelectorAll('.page'); + if (allPages.length % 2 == 1) { + this.div.appendChild(Pagination.createPages(1, false, this.pageCounter.selector, this.columns)); + } +}; + +Pagination.flowObject.prototype.addPagesLoop = function (pages) { + + if ('undefined' === typeof (pages)) { + this.totalPages += this.bulkPagesToAdd; + this.div.appendChild(Pagination.createPages(this.bulkPagesToAdd, this.name, this.pageCounter.selector, this.columns)); + this.bulkPagesToAdd = Math.floor(this.bulkPagesToAdd * Pagination.config.pagesToAddIncrementRatio); + } else { + this.totalPages += pages; + this.div.appendChild(Pagination.createPages(pages, this.name, this.pageCounter.selector, this.columns)); + } + this.addOrRemovePages(pages); +}; + + +Pagination.flowObject.prototype.addOrRemovePages = function (pages) { + + if ((this.namedFlow.overset) && (this.rawdiv.innerText.length > 0)) { + this.pageCounter.needsUpdate = true; + this.redoPages = true; + this.addPagesLoop(pages); + } else if ((this.namedFlow.firstEmptyRegionIndex != -1) && ((this.totalPages * this.columns - this.namedFlow.firstEmptyRegionIndex) > this.columns)) { + this.redoPages = true; + this.removeExcessPages(pages); + } else if (this.redoPages) { + this.redoPages = false; + if (Pagination.config.alwaysEven) { + this.makeEvenPages(); + } + if (this.name != 'frontmatter') { + document.body.dispatchEvent(Pagination.events.bodyLayoutUpdated); + } + } +}; + + +Pagination.flowObject.prototype.removeExcessPages = function (pages) { + + var allPages = this.div.querySelectorAll('.page'); + + for (var i = (Math.ceil(this.namedFlow.firstEmptyRegionIndex / this.columns)); i < allPages.length; i++) { + this.div.removeChild(allPages[i]); + this.totalPages--; + } + this.addOrRemovePages(pages); +}; + + +Pagination.flowObject.prototype.setupReflow = function () { + var flowObject = this; + + var checkOverset = function () { + // Something has changed in the contents of this flow. Check if the overset has changed, and if it has, emit a pageLayoutUpdate event. + if ((flowObject.namedFlow.overset !== flowObject.overset) || (flowObject.namedFlow.firstEmptyRegionIndex !== flowObject.firstEmptyRegionIndex)) { + flowObject.overset = flowObject.namedFlow.overset; + flowObject.firstEmptyRegionIndex = flowObject.namedFlow.firstEmptyRegionIndex; + flowObject.namedFlow.dispatchEvent(Pagination.events.pageLayoutUpdate); + } + } + this.namedFlow.addEventListener('webkitRegionLayoutUpdate', checkOverset); + + var reFlow = function () { + // The page layout has changed. Reflow by adding pages one by one. + flowObject.addOrRemovePages(1); + flowObject.pageCounter.numberPages(); + }; + this.namedFlow.addEventListener('pageLayoutUpdated', reFlow); + +}; + + + +Pagination.initiate(); + +if (Pagination.config.autoStart === true) { + // Hook Pagination.autoStartInitiator to document loading stage if + document.addEventListener("readystatechange", Pagination.autoStartInitiator); +} diff --git a/examples/openspending/content/resources/handbook/vendor/sweet-justice.min.js b/examples/openspending/content/resources/handbook/vendor/sweet-justice.min.js new file mode 100644 index 00000000..c83e5a5d --- /dev/null +++ b/examples/openspending/content/resources/handbook/vendor/sweet-justice.min.js @@ -0,0 +1 @@ +!function(){var h=6;var p={code:true,pre:true,abbr:true};function b(v){var c=v.childNodes;for(var e=0;e +
    + +
    +
    + {{body}} +
    +