[openspending-dev] UK Local Govt: Personal data - suppression lists

[Ian Makgill]

We're starting work on some routines to suppress personal data appearing in
UK Council spend data statements.

This is the approach we're taking:

1. String search - looking for common names, salutations etc
2. Pattern matching - looking for common patterns, e.g. a single uppercase
letter followed by a full stop and a space frequently indicates a name.
3. Velocity - looking for common patterns in payments, for example citizens
in receipt of Social Care Direct Payment will be paid monthly on the same
date
4. Value - The highest value Direct Payment is around £1,500 per week, we
search for payments below £10,000 to help us narrow our efforts

The bottom line is that you have to put human eyes on each transaction that
looks like it might be an individual.

To avoid duplication, we're interested in working with the OpenSpending
community on some suppression lists for named individuals.

Some immediate questions spring to mind:

1. How do we host? Since the data is private, Github probably isn't the
best repository.

2. What data to include in the list? We suggest the following:
Recipient String,
Buyer Name,
Source file name,
Source URL,
Date of transaction (if available),
Details of the org (or individual) who created the record.

3. Validation and moderation? Personal knowledge (see below) is vital here,
how do we ensure that we don't incorrectly suppress payments?

Some payments to individuals are quite legitimate, for instance Harrow
Council recently paid £1,076 to Karen Buck, the MP for Westminster North,
this was recorded simply as Karen Buck. Unless we'd known that Karen Buck
was Harrow's local MP we'd have marked this payment as personal. It is also
common for Barristers and Doctors to use their names when billing for work.

We'd be interested to see how the community might be able to find a
solution to this problem.

Regards,

Ian Makgill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20130214/aec7ce3a/attachment.html>