A few weeks ago, I hit the submit button to send off our Feasibility Study to the Department of Innovation, successfully concluding the first phase of the Business Research Innovation Initiative.
We had been invited to test the feasibility of our proposed platform for AUSTRAC (the Australian Transaction Reporting Centre) and ACIC (the Australian Criminal Intelligence Commission) to track which of their intelligence reports were being read, and by whom.
Specifically, we were exploring the feasibility of a machine learning/natural language processing approach to help give these agencies better visibility of the value of their intelligence reports.
For those who don’t know, AUSTRAC is the Australian government’s specialist financial intelligence unit and the ACIC is the Commomwealth agency supporting law enforcement intelligence efforts for serious and organized crime.
These agencies are looking to snare the bad guys – drug traffickers, fraudsters, money launderers, and so on. Every day, these agencies are analysing, consolidating and publishing intelligence for their partners – agencies like the Australian Federal Police, the Attorney-General’s Department, and many others.
The trouble is, AUSTRAC and ACIC don’t really have the technical capability to understand which intelligence reports were useful to their partners, and which were not. Asking partners to provide manual feedback is unreliable, as few people have the time or the incentive to respond in a systematic fashion.
The way we saw it, there are two options for attacking the problem.
The first is to squeeze as much insight as possible from conventional metrics – i.e. metrics that don’t require the user to do anything.
There is obviously some correlation between the number of times a report is opened, and its broader value to the community. There will still be a bit of noise in these metrics – some reports are opened by mistake, others might turn out to be irrelevant after the user finishes reading, and so on. But we speculated that this noise would be relatively evenly distributed throughout the entire library of intelligence reports.
The key, therefore, would be to find some way to meaningfully aggregate metrics to show trends in interest over time – the agencies obviously don’t want to manually review every single intelligence report they ever published to understand what their partners are interested in. That’s akin to finding trending news by reading every article on every single news site on the internet – not a great use of resources. We needed to ensure that the agencies were able to see, at a glance, what types of reports were of interest to their partners.
The second option for attacking the problem is just as simple – find some way to encourage users to provide their own feedback. We want end-users to provide as much manual feedback as possible, whether rating reports on a 5-star scale, writing case summaries, or through some other mechanism.
Unfortunately, users won’t provide this feedback simply by being asked – that’s the whole reason we were involved in the first place. I refer to this as the “lazy user problem” – slightly unfair to describe users this way, but at least it gets the the point across.
It was important to find some way to encourage users to participate – find a good incentive so that users want to provide feedback.
We figured a good carrot-and-stick approach would be provide users with a list of “More like this” report recommendations. Since users find these valuable, the system could stop providing these recommendations once feedback drops below a certain threshold. Users could regain access with a few mouse clicks, by voting on valuable reports or giving more detailed written feedback.
We could also deal with the lazy user problem by circumventing it completely. If the system could bypass the user and automatically analyse the user’s inbox and file history, the system could fairly easily detect which intelligence reports were useful to that user. This may be impractical given certain security/implementation considerations, but we took a similar approach to prompt users to publish feedback by matching their local data against their report access history.
To achieve these objectives, we blended two conventional NLP techniques to build and test the model on a sample dataset.
The first is “Named Entity Extraction”, which basically means extracting the names people, places organizations, and so on. Examples of “named entities” in the context of AUSTRAC/ACIC intelligence reports, this might include the names of targets under investigation, criminal organizations, drug names, and so on.
The second is “Topic Modelling”, which, as the name suggests, basically refers to determining what “topic” a particular document relates to. An AUSTRAC/ACIC intelligence report, for example, might relate to a particular region, drug, or crime.
For now, it’s enough to say that we successfully showed that it was possible to use these techniques not only to aggregate access statistics, but also as a recommendation system to incentivize/prompt users to provide feedback on intelligence reports.
Over the coming days and weeks, I’ll be going into further detail as to how this can actually be done. It’s fascinating to peek under the hood and see how it all works, and I look forward to sharing this with you against the background on an actual, real-life problem.
Thanks again to the Department of Industry, Innovation and Science, AUSTRAC and the ACIC for their support.