Conference season online

October has always been a super busy month for me. I’m usually starting a new OU module and travelling around speaking at conferences and meetups, all while doing my day job, spending time with my family and enjoying my hobbies. Sometimes I’ve not got the balance right! 2019 I remember was particularly hectic. I optimistically submitted conference sessions at the start of the year on a variety of different topics and, as the year went on I was invited to speak at various meetups in the UK and even stepped in to do some last minute presentations where other speakers had dropped out. This time last year I had just finished 8 weeks where I had a week’s holiday, spoken at 5 conferences, 2 breakfast briefings and 8 meet ups, all of which were on slightly different topics!

I really enjoy speaking at these events, otherwise I simply wouldn’t do them! As an attendee I get to learn from my peers and be inspired by steps forward in areas that I just don’t have time to keep up to date on. As a speaker, I get to pass on some of the things I’ve learned over the years in what I hope is an entertaining way, and I always love the conversations after the talks.

This year has, inevitably, been very different. February was my first event, where I spoke at The European Information Security Summit in London on the risks that Deep Fakes pose to the security sector. I spoke to a lot of security professionals at the event who were unaware of how AI was progressing in both voice and face cloning. As an attendee, I learned a lot about the state of security in many of the systems we take for granted. If you can justify the time and cost, attending a conference outside of your area of expertise can be incredibly informative.

A mere few weeks later, and I had several sessions at Big Data and AI World. I had a panel session with the amazing Sue Daley and Vitaliy Yuryev on why basics are often overlooked in data projects, followed a few hours later by my main presentation on learning from projects that go wrong. This was the 12th of March. While the organisers were doing everything they could, practically all the international speakers and attendees had decided not to attend. The sessions were reorganised to prevent large gaps in the program and many of the sessions I had been personally looking forward to were no longer happening. I really enjoyed both my sessions and got some great questions after them, but it was clear that people were nervous about the crowds and conferences and meetups would be on hold from that point onwards.

As I headed home on the train that afternoon I knew I wouldn’t be back in London for a while. My company was considering a trial of homeworking for a few days a week1, but I’d already decided to swap to home working for the foreseeable future and told my team to do the same if they wanted. My team had been at the conference with me and I didn’t realise then that it would be the last time I’d see them2.

March and April would normally be the time that I would be submitting keynote suggestions for the Autumn conference season and spending my evenings talking to University students at meetups and I really missed those interactions.

While I was interviewed over the summer (Humans of AI, Agile Data Science), I really did miss the chance to interact with a wider audience. You can’t respond to questions in a pre-recorded video.

I was delighted when Barclays Eagle Labs asked me if I would rerun a talk on Deep Fakes that I had given in person late in 2019, as a series of three online events. Despite the strangeness of talking into a camera without the feedback of the audience’s faces and the ever present anxiety that one of my neighbours would start noisily doing DIY during the session3, it was great to see so many people take 30 minutes out of their day for three consecutive weeks to learn. After the final session, I got a lot of messages from people who had made their own fakes and really understood both the positive and negative aspects of the technology and thanking me for making it accessible. It’s this type of interaction that makes these events worthwhile. Sadly these sessions were not recorded, but the slides are on my slideshare (Part 1, Part 2 and Part 3) and a variation of the talk that I gave at Tech Exeter in 2019 is available on sitepoint.

At the end of September, one of the events that was cancelled from March was resurrected as an online event sponsored by DevelopHer. I had 5 minutes, which is both an eternity (if you’ve ever heard Just a Minute) and the blink of an eye (if you have more than a single thing you want to say)! I managed to condense the 25 talk on getting into Data Science and AI into (just over) 5 minutes alongside an amazing line up of other women in AI.

What really stood out to me from this event is how many people attended who may not otherwise have been able to go to an in person meetup. Not everyone has the luxury of being able to stay late after work, or travel in for these events, or may not want to even if they could. One of the huge benefits of everything moving online is that it has made many of these events far more accessible, and I hope that this continues in some form.

Post by Bethan Reeves watching my talk at home in comfort 😀

Last week I spoke at the online version of one of my favourite conferences, Minds Mastering Machines. The invite advertised me as one of their veteran speakers :D. I’ve done some heavily technical talks at their event over the past few years, but for 2020 I decided to be a bit lighter and given world events I’m glad I did. One of the things I’ve noticed in all the projects I’ve led, advised on, or done due diligence for, is that testing never seems to be a priority for data science and AI. This is something that drives me crazy so I thought I’d approach it in a light hearted way and try and convert the attendees to testing thinking with a talk titled: Your testing sucks – what should you be doing? I paired seven best practises of testing thinking alongside examples (mostly) from spacecraft. I think it went down well and hopefully it was memorably enough to make people want to make time for testing by remembering the various missions.

My presentation from MCubed. Don’t coerce your data.

While I’ve nothing else planned for this year or even 2021, I intend to speak at more conferences. Even when large gatherings are safe again, I hope that there will still be online streams for those that cannot attend. Let’s keep tech accessible.

A diagnostic tale of docker

twenty sided die showing common excuses for developers not to fix problems, the top of the die shows "Can't reproduce"
Developer d20 gives the answer 🙂 (from Pretend Store)

If you’ve been to any of my technical talks over the past year or so then you’ll know I’m a huge advocate for running AI models as api services within docker containers and using services like cloud formation to give scalability. One of the issues with this is that when you get problems in production they can be difficult to trace. Methodical diagnostics of code rather than data is a skill that is not that common in the AI community and something that comes with experience. Here’s a breakdown of one of these types of problems, the diagnostics to find the cause and the eventual fix, all of which you’re going to need to know if you want to use these types of services.

Read more

Let’s talk about testing

One of the things that I find I have to teach data scientists and ML researchers almost universally is understanding how to test their own code. Too often it’s all about testing the results and not enough about the code. I’ve been saying for a while that a lack of proper testing can trip you up and recently we saw a paper that rippled through academia about a “bug” in some code that everyone used…

A Code Glitch May Have Caused Errors In More Than 100 Published Studies

https://www.vice.com/en_us/article/zmjwda/a-code-glitch-may-have-caused-errors-in-more-than-100-published-studies

The short version of this is that back in 2014, a python protocol was released for calculating molecule structure through NMR shifts1 and many other labs have been using this script over the past 5 years.

Continue reading Let’s talk about testing

Fooling AI and transparency – testing and honesty is critical

The effect of digitally changing an image on a classifier. Can you tell the difference between the pictures? Image from Brendel et al 2017.

If you follow my posts on AI (here and on other sites) then you’ll know that I’m a big believer on ensuring that AI models are thoroughly tested and that their accuracy, precision and recall are clearly identified.  Indeed, my submission to the Science and Technology select committee earlier this year highlighted this need, even though the algorithms themselves may never be transparent.  It was not a surprise in the slightest that a paper has been released on tricking “black box” commercial AI into misclassification with minimal effort. Continue reading Fooling AI and transparency – testing and honesty is critical

Why are data scientists so bad at science?

Do you check your inputs?

It’s rare that I am intentionally provocative in my post titles, but I’d really like you to think about this one. I’ve known and worked with a lot of people who work with data over the years, many of who call themselves data scientists and many who do the role of a data scientist but by another name1. One thing that worries me when they talk about their work is an absence of scientific rigour and this is a huge problem, and one I’ve talked about before.

The results that data scientists produce are becoming increasingly important in our lives; from determining what adverts we see to how we are treated by financial institutions or governments. These results can have direct impact on people’s lives and we have a moral and ethical obligation to ensure that they are correct. Continue reading Why are data scientists so bad at science?

Testing applications

It's true. Image from Andy Glover http://cartoontester.blogspot.com/
It’s true. Image from Andy Glover http://cartoontester.blogspot.com/

As part of a few hours catching up on machine learning conference videos, I found this great talk on what can go wrong with machine recommendations and how testing can be improved.  Evan gives some great examples of where the algorithms can give unintended outputs.  In some cases this is an emergent property of correlations in the data, and in others, it’s down to missing examples in the test set.  While the talk is entertaining and shows some very important examples, it made me realise something that has been missing.  The machine learning community, having grown out of academia, does not have the same rigour of developmental processes as standard software development.

Regardless of the type of machine learning employed, testing and quantification of results is all down to the test set that is used.  Accuracy against the test set, simpler architectures, fewer training examples are the focal points.  If the test set is lacking, this is not uncovered as part of the process.  Models that show high precision and recall can often fail when “real” data is passed through them.  Sometimes in spectacular ways as outlined in Evan’s talk:  adjusting pricing for Mac users, Amazon recommending inappropriate products or Facebook’s misclassification of people.  These problems are either solved with manual hacks after the algorithms have run or by adding specific issues to the test set.  While there are businesses that take the same approach with their software, they are thankfully few and far between and most companies now use some form of continuous integration, automated testing and then rigorous manual testing.

The only part of this process that will truly solve the problem is the addition of rigorous manual testing by professional testers.  Good testers are very hard to find, in some respect harder than it is to find good developers.  Testing is often seen as a second class profession to development and I feel this is really unfair.  Good testers can understand the application they are testing on multiple levels, create the automated functional tests and make sure that everything you expect to work, works.  But they also know how to stress an application – push it well beyond what it was designed to do, just to see whether these cases will be handled.  What assumptions were made that can be challenged.  A really good tester will see deficiencies in test sets and think “what happens if…”, they’ll sneak the bizarre examples in for the challenge.

One of the most difficult things about being a tester in the machine learning space is that in order to understand all the ways in which things can go wrong, you do need some appreciation of how the underlying system works, rather than a complete black box.  Knowing that most vision networks look for edges would prompt a good tester to throw in random patterns, from animal prints to static noise.  A good tester would look of examples not covered by the test set and make sure the negatives far outweigh the samples the developers used to create the model.

So where are all the specialist testers for machine learning?  I think the industry really needs them before we have (any more) decision engines in our lives that have hidden issues waiting to emerge…