datahub/examples/openspending/content/blog/2013-03-13-Launching-the-Aid-Transparency-Tracker.mdown

---
authors:
- mark
redirect_from: /2013/03/Launching-the-Aid-Transparency-Tracker/
title: Publish What You Fund Launches Aid Transparency Tracker
---

Publish What You Fund has undertaken some initial [analysis of aid donors’ plans](http://tracker.publishwhatyoufund.org/) to publish to the [IATI](http://www.aidtransparency.net/) component of the agreed [common standard](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) for aid information. Here, [Mark Brough](https://twitter.com/mark_brough) explains the process they went through to take a series of Excel files, convert them into a format suitable for analysis, and come to some overall conclusions.

The short version: check out the [Aid Transparency Tracker](http://tracker.publishwhatyoufund.org/).

###Why we did this
At the 4th High Level Forum on Aid Effectiveness in Busan, South Korea, in November 2011, the world’s aid donors agreed a “common, open standard” for publishing aid information. Donors also agreed in Busan to publish “implementation schedules” explaining in detail how and when they would meet this commitment.

The implementation schedules are like a forward-looking calendar, explaining when donors plan to publish specific pieces of data, like results, project documents, and conditions, as well as transaction-level spending data. They also explain whether the donor will be publishing under an open license (public domain or attribution-only) and whether they will be republishing every quarter as a minimum frequency of publication – both are required for IATI compliance. These implementation schedules were published on the [OECD/DAC’s common standard website](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm), but in several different formats, which required a detailed look at each donor’s schedule, as well as interpreting them when donors have completed them in different ways or understood the various options differently.

Pulling all the schedules into a single application allows us to [assess donors’ overall ambition](http://tracker.publishwhatyoufund.org/organisations/); [compare fields across schedules](http://tracker.publishwhatyoufund.org/fields/); show the publication of [fields over time from different donors](http://tracker.publishwhatyoufund.org/timeline/); and provide [CSV](http://tracker.publishwhatyoufund.org/organisations/GB-1.csv), [JSON](http://tracker.publishwhatyoufund.org/api/publishers/GB-1/)/[JSONP](http://tracker.publishwhatyoufund.org/api/publishers/GB-1/?callback=callback) and [iCal feeds](http://tracker.publishwhatyoufund.org/organisations/GB-1.ics) for each donor.

###The original implementation schedules
The implementation schedules were published in individual Excel files, containing three main sheets: general, agency, and activity-level information.

<br><strong>General</strong><br>
Approach to publication: includes timeliness and frequency, licence, initial publication dates, scope of publication.<br>
<table border="" cellpadding="0" cellspacing="1px">
  <tr>
    <td>
<img alt="" src="http://tracker.publishwhatyoufund.org/static/img/sweden1.png" title="" class="inline-img" width="300" height="246" />
 </td>
    <td>
</tr>
</table>

<br><strong>Agency</strong><br>
Agency-level publication: includes country budgets, organisation documents.<br>
<img alt="" src="http://tracker.publishwhatyoufund.org/static/img/sweden2.png" title="" class="inline-img" width="300" height="246" />
    <td>
</tr>
</table>

<br><strong>Activity</strong><br>
Activity-level publication: includes information about when the organisation will (or won’t) be compliant with each field in the standard.<br>
</td>
<table border="" cellpadding="0" cellspacing="1px">
  <tr>
    <td>
<img alt="" src="http://tracker.publishwhatyoufund.org/static/img/sweden3.png" title="" class="inline-img" width="300" height="246" />
    <td>
  </tr>
</table>
<br>
###Templates
While there is a template for the common standard implementation schedules, several different templates exist, and donors further added to this complexity by modifying the templates, changing options, and adding and deleting rows – in all, we counted eleven different versions. Importing the schedules proved difficult because of this. In addition, some donors published their Excel-based templates in PDF format, which made it impossible to automatically parse them. In these cases, it was necessary to copy and paste the same data into new spreadsheets. While the data was thoroughly checked, it is possible that some human error will have resulted. Some interpretation was necessary to ensure consistency and comparability across the schedules.

###Importing
The schedules were automatically parsed and imported into this application. Publish What You Fund staff then checked the resulting data to ensure that it had been parsed correctly and that it made sense.

<br><strong>Select a file</strong><br>
The user can select a schedule from any publicly-accessible URL. This could be the [OECD/DAC common standard](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) website, the [IATI website](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm), or the donor’s own website. Where the original schedules could not be automatically parsed (e.g. because they were in PDF), a new spreadsheet was created by Publish What You Fund using the information included in the original schedule.
</td>
<table border="" cellpadding="0" cellspacing="1px">
  <tr>
    <td>
<img alt="" src="http://tracker.publishwhatyoufund.org/static/img/import.png" title="" class="inline-img" width="300" height="246" />
    <td>
  </tr>
</table>

<br><strong>Check fields</strong><br>
The schedule was successfully parsed and is presented to the user to check and correct, make sure that all information has been parsed correctly, ensure that compliance status is consistent with notes, and score for partially compliant fields if the publisher has understated their publication relative to other donors.
The user can select a schedule from any publicly-accessible URL. This could be the OECD/DAC common standard website, the IATI website, or the donor’s own website. Where the original schedules could not be automatically parsed (e.g. because they were in PDF), a new spreadsheet was created by Publish What You Fund using the information included in the original schedule.<br>
</td>
<table border="" cellpadding="0" cellspacing="1px">
  <tr>
    <td>
<img alt="" src="http://tracker.publishwhatyoufund.org/static/img/import3.png" title="" class="inline-img" width="300" height="246" />
    <td>
  </tr>
</table>

###Openness begets openness
None of this would have been possible without:<br>
1) A range of great open-source tools:
- [iati-implementationxml](https://github.com/Bjwebb/iati-implementationxml) (originally converted Excel IATI implementation schedules to XML)
- OKFN’s [ReclineJS](http://reclinejs.com/) (for the graphs and timeline; which itself relies on a range of great open source software)
- [Bootstrap](http://twitter.github.com/bootstrap/), [JQuery](http://jquery.com/), and [JQuery Tablesorter](http://tablesorter.com/docs/) (for the front-end)
- [Python](http://python.org), [Flask](http://flask.pocoo.org/), and a range of libraries (for the back-end)</li></ul><br>
2) [Implementation schedules](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm) released by donors as part of the common standard, transparently outlining in detail their specific commitments to publish more and better data.

But, we’ve also fed back: iati-implementationxml has been expanded to add compatibility with the common standard formats of implementation schedules; to detect the format of different schedules; and to function as a module that can be imported. We provided a couple of pull request for some small bugs with ReclineJS, and of course, we’re releasing all of our own code as well (although, it could definitely do with some tidying up…).

###What’s next?
Now that we’ve looked at what information different donors are committing to publish, there are two main steps for us:<br>

1) <strong>Doing it again:</strong> we want to check the assumptions and interpretations we’ve made with donors to ensure that they’re accurate, as well as encouraging all donors to be more ambitious about the information they’re planning to publish. As part of this, we’ll also aim to iron out the problems in the schedule.<br>
2) <strong>Looking at publication:</strong> now that we have better data on what donors are committing to do, the question arises: what are they actually publishing, and are they meeting their commitments? We’ll be working on that throughout the year and will release this analysis in October. Until then, all of our code is of course being developed as an [open-source project on Github](https://github.com/markbrough/IATI-Data-Quality).
<br>
<br>We’d love your feedback, so please get in touch:<br>
[email](mailto:info@publishwhatyoufund.org)<br>
twitter: [@mark_brough](https://twitter.com/mark_brough) or [@aidtransparency](https://twitter.com/aidtransparency)

###Links
[Aid Transparency Tracker](http://tracker.publishwhatyoufund.org/)<br>
[Source code](https://github.com/Bjwebb/iati-implementationxml)<br>
Licenses: [AGPL v3](http://www.gnu.org/licenses/agpl-3.0.html) (code); [PDDL](http://opendatacommons.org/licenses/pddl/) (data); [CC-BY](http://creativecommons.org/licenses/by/3.0/) (report and analysis)<br>
[IATI (International Aid Transparency Initiative)](http://www.aidtransparency.net/)<br>
[OECD/DAC Common Standard website](http://www.oecd.org/dac/aid-architecture/acommonstandard.htm)