Testing applications

It's true. Image from Andy Glover http://cartoontester.blogspot.com/
It’s true. Image from Andy Glover http://cartoontester.blogspot.com/

As part of a few hours catching up on machine learning conference videos, I found this great talk on what can go wrong with machine recommendations and how testing can be improved.  Evan gives some great examples of where the algorithms can give unintended outputs.  In some cases this is an emergent property of correlations in the data, and in others, it’s down to missing examples in the test set.  While the talk is entertaining and shows some very important examples, it made me realise something that has been missing.  The machine learning community, having grown out of academia, does not have the same rigour of developmental processes as standard software development.

Regardless of the type of machine learning employed, testing and quantification of results is all down to the test set that is used.  Accuracy against the test set, simpler architectures, fewer training examples are the focal points.  If the test set is lacking, this is not uncovered as part of the process.  Models that show high precision and recall can often fail when “real” data is passed through them.  Sometimes in spectacular ways as outlined in Evan’s talk:  adjusting pricing for Mac users, Amazon recommending inappropriate products or Facebook’s misclassification of people.  These problems are either solved with manual hacks after the algorithms have run or by adding specific issues to the test set.  While there are businesses that take the same approach with their software, they are thankfully few and far between and most companies now use some form of continuous integration, automated testing and then rigorous manual testing.

The only part of this process that will truly solve the problem is the addition of rigorous manual testing by professional testers.  Good testers are very hard to find, in some respect harder than it is to find good developers.  Testing is often seen as a second class profession to development and I feel this is really unfair.  Good testers can understand the application they are testing on multiple levels, create the automated functional tests and make sure that everything you expect to work, works.  But they also know how to stress an application – push it well beyond what it was designed to do, just to see whether these cases will be handled.  What assumptions were made that can be challenged.  A really good tester will see deficiencies in test sets and think “what happens if…”, they’ll sneak the bizarre examples in for the challenge.

One of the most difficult things about being a tester in the machine learning space is that in order to understand all the ways in which things can go wrong, you do need some appreciation of how the underlying system works, rather than a complete black box.  Knowing that most vision networks look for edges would prompt a good tester to throw in random patterns, from animal prints to static noise.  A good tester would look of examples not covered by the test set and make sure the negatives far outweigh the samples the developers used to create the model.

So where are all the specialist testers for machine learning?  I think the industry really needs them before we have (any more) decision engines in our lives that have hidden issues waiting to emerge…

 

Evolving Machines

I, for one, welcome our new metal overlords ;)
I, for one, welcome our new metal overlords 😉

Following my post on AI for understanding ambiguity, I got into a discussion with a friend covering the limitations of AI if we only try to emulate ourselves.  His premise was that we know so little about how our brains actually enable us to have our rich independent thoughts that if we constrain AI to what we observe, an ability to converse in our native language and perform tasks that we can do with higher precision, then we will completely limit their potential.  I had a similar conversation in the summer of 2015 while at the start-up company I joined1– we spent a whole day2 discussing whether in 100 years’ time the only jobs that humans would do would be to code the robots.  While the technological revolution is changing how we live and work, and yes it will remove some jobs and create others just like the industrial revolution did and ongoing machine automation has been doing, there will always be a rich variety of new roles that require our unique skills and imagination, our ability to adapt and look beyond what we know. Continue reading Evolving Machines

AI for understanding ambiguity

Please do not park bicycles against these railings as they may be removed - the railings or the bikes? Understanding the meaning is easy for us, harder for machines
Please do not park bicycles against these railings as they may be removed – the railings or the bikes? Understanding the meaning is easy for us, harder for machines

Last year I wrote a post on whether machines could ever think1.  Recently, in addition to all the general chatbot competitions, there has been a new type of test for deeper contextual understanding rather than the dumb and obvious meanings of words.  English2 has a rich variety of meanings of words with the primary as the most common and then secondary and tertiary meanings further down in the dictionary.  It’s probably been a while since you last sat down and read a dictionary, or even used an online one other than to find a synonym, antonym or check your spelling3 but as humans we rely mostly on our vocabulary and context that we’ve picked up from education and experience.

Continue reading AI for understanding ambiguity

3D Printer part 16: top plate and roller holder

At the end of my last post in this series, we had added the power module and fan.  This post focuses on the top plates and roller holder covering issues 60 – 63 of 3D Create and Print by Eaglemoss Technology.  If you’ve skipped a part of this series you can start from the beginning, including details of the Vector 3 printer I’m building on my 3D printer page.

This drop comes with another much needed binder for the magazines, starting to get us back into phase with the right number of binders to loose magazines.  With the plate covers we enclose the wires and finally start the process of building the components for the top of the printer. Continue reading 3D Printer part 16: top plate and roller holder

3D Printer part 15: Power module and fan

At the end of my last post in this series, we had added the USB, microcontroller and motor boards.  This post focuses on the power module and fan covering issues 56-59 of 3D Create and Print by Eaglemoss Technology.  If you’ve skipped a part of this series you can start from the beginning, including details of the Vector 3 printer I’m building on my 3D printer page. Continue reading 3D Printer part 15: Power module and fan

3D Printer part 14: USB, microcontroller and motor boards

At the end of my last post in this series, we had added the power switch and control unit covers and the look of the main part of the printer was starting to come together.  This post focuses on the USB, microcontroller and motor boards covering issues 52-55 of 3D Create and Print by Eaglemoss Technology.  If you’ve skipped a part of this series you can start from the beginning, including details of the Vector 3 printer I’m building on my 3D printer page.

Due to some of my OU studies and work, I figure I’m now 6 months behind in this build so please accept my apologies if you’ve been waiting on my posts to do your own builds. So, when I finally opened the pack, I initially panicked to see that there were two issue 55 parts that came in my pack.  On checking the magazines, they had different part numbers but were identical in every respect as far as I could tell from the magazine.  As such, I have used these parts as supplied.  If this turns out to be a problem I’m sure I’ll find out later… Continue reading 3D Printer part 14: USB, microcontroller and motor boards

Rework DL Boston 2016 – Day 2

Me, networking at breakfast
Me, networking at breakfast

This is a summary of day 2 of ReWork Deep Learning summit 2016 that took place in Boston, May 12-13th.  If you want to read the summary of day 1 then you can read my notes here. Continue reading Rework DL Boston 2016 – Day 2

ReWork DL Boston 2016 – Day 1

brainLast year, I blogged about the rework Deep Learning conference in Boston and, being here for the second year in a row, I thought I’d do the same.  Here’s the summary of day 1.

The day started with a great intro from Jana Eggers with a positive message about nurturing this AI baby that is being created rather than the doomsday scenario that is regularly spouted.  We are a collaborative discipline of academia and industry and we can focus on how we use this for good. Continue reading ReWork DL Boston 2016 – Day 1

Surface Pro: how I use it – a review

Surface Pro 3I first used the Surface Pro 3 on my trip to Boston to take notes at ReWorkDL rather than scribbling on bits of paper or taking a full laptop and found it to be a great replacement for an A4 notebook, but didn’t really use it to its full potential.  At the start of November, I joined a new company and I’ve been using the Surface exclusively for all my note taking, as well as for studying for my OU Maths modules.

With the recent release of the Surface 4, there may be people wondering if they’re worth it, and what use they’d get out of it.  There are plenty of technical reviews around so I’d suggest using those as a starting point, and if you’re headed out to the sales, you might find my experiences helpful.   Continue reading Surface Pro: how I use it – a review

Girls Can Code – episode 2 thoughts

Alice Levine showing that Girls Can Code
Alice Levine showing that Girls Can Code

After somewhat mixed reviews of last week’s episode I was interested to see whether episode 2 of Girls Can Code had any more emphasis on the coding side.

It started with a comment that the girls were building a tech business rather than actually learning to code themselves.  This justified one of the major criticisms of the show – that it was nothing about coding.  While I’m sure that this was all fixed months ago, I did wonder if the voice over for the start of the show had been rerecorded after the response to the first episode. Continue reading Girls Can Code – episode 2 thoughts