My DeepLabCut journey
I saw the DeepLabCut paper when it came out. I can’t say I read it, it was way too technical for me at the time, but I saw the videos that showed the tracking, and I was blown away. I knew immediately that if I managed to get this working for me, it would open up a world of opportunities. Tracking arbitrary points from video of rough quality would mean that I could do experiments in the field under natural conditions where it would be impossible to have a nice clean background free from clutter. Till now I had mostly relied on manually tracking insects and spiders, using either a custom built Matlab based tracking program (Digilite) or MTrackJ in ImageJ/FIJI. While this was perfectly fine for short videos and for a couple of points per animal, it was very laborious and tedious, and very hard to scale. It limits the kind of questions one can ask.
I was in Bangalore, India for a couple of months in 2019, and I went to visit Maria Thaker at the CES, IISc. While chatting about a potential collaboration, I mentioned the existence of DLC, and was told that a neighbouring lab was looking into using machine learning for tracking. Generally, one of my biggest problem working where I am is knowing people who have technical expertise in computing. So I leaped at the chance to learn more about it. I met the lab head Vishu Guttal and asked about their efforts in automatic tracking. One of Vishu’s students had tried out DLC but told me that they later went for an in house solution for some arcane reason that I no longer remember. But Vishu very generously put me in touch with a student, Arun, who he said could help me get started in this new world.
The first problem was that I use a Mac. All machine learning programs make heavy use of NVidia GPUs. Nvidia and Apple no longer get along, and so to take full advantage of the machine learning stuff, I had to either use a Windows or a Linux system, neither of which was appealing. However, Deeplabcut had a CPU version which could theoretically run on a Mac, but that was just to try out the software since it would be painfully slow. Arun first tried to set up the system on the lab’s Linux machine which had the appropriate GPU and I followed along on my mac. It was slow going, involving a complete formatting of the system which alarmed me quite a lot, but he seemed unfazed. He managed to get it working on the lab machine, but I was running out of time, and needed to head back to Mexico. I tried to get it running on my machine but there was a bug that only affected my operating system, and the only way out was to upgrade my OS, which I couldn’t do since it would affect other software that I used frequently.
I returned to Mexico with a better understanding of the challenges involved. Back at the office, I updated the OS on my office Mac so I wouldn’t have the same bug. After countless hours of googling and python wrangling I managed to install the software. I followed the tracking instructions and found, as expected, that the system would need several days before I could even find out whether the software worked. This was not going to work, I needed to invest in a GPU.
Of course I had no idea where to begin with the huge range of GPU options. I had some money in a grant to buy a laptop, so I wrote to the funding agency to change it to a small PC. I went with the Intel NUC, a gaming PC complete with LED skull and all. Then I bought the exact same GPU that the Deeplabcut people used. I then found out that I needed to buy a box to put the GPU in, which cost as much as the GPU itself! The PC came with windows, but I asked our IT guy at the institute to format everything and install Ubuntu on it. Thinking it would be simpler. hah.
The hardware was setup, but I couldn’t find time to try setting up DLC. Meanwhile, we started using this machine in the lab to do manual and semi automatic tracking of kleptoparasitic spider movement using the program Tracker. Since the system was quite powerful it was easier to run things here than on the lab Macs, which were really old by now. I decided to give Deeplabcut another go in 2020 instead.
In March 2020, we went into lockdown due to the Covid-19 virus. In the beginning, I, like many others, thought that we would be working from home for a couple of months and I thought I would just manage with my now ancient laptop, but as it went on, it was apparent that working from home was here to stay. I brought my main desktop mac home, and since there was no one using the Linux machine, I brought that home as well. At this point, I started converting the NUC into a dedicated DeepLabCut machine. I enlisted the help of my collaborator Horacio Tapia McClung, who conveniently lives very close to me, so he could come over and get it working. It was a lot of effort to convert the Intel NUC from a gaming machine to a Machine learning device. I wish I could say what this effort involved but it was way over my head. I think the hardest part was to engage the GPU. 
Once this was done, I started installing the DeepLabCut software. This was a blur of intense googling, reading up basic commands in Ubuntu, python etc etc. Even the simplest thing took me ages to figure out. After several days of this, I managed to get it working. It was such a relief and pleasure to see the launch of the DLC GUI. Even then it wasn’t smooth sailing, I kept having weird problems with Ubuntu, everything seemed so fragile. I couldn’t login and had to figure out how to hack into my own account. The display manager had conflicts, etc etc, it was endless. After a couple of weeks of this, I finally reached a state where everything was stable: the GPU was activated, the OS was fine, DeepLabCut was working and I finally could start testing it on my own videos. 
In May 2020, there was a huge hailstorm. The intel NUC was set up in my studio. There was a channel on the roof that took rain water down to the garden. The channel got blocked somehow, we think with hailstones, and the water filled up the channel, and the pressure forced a hole in the surrounding wall. Which led directly into the studio. So suddenly the studio had an indoor waterfall, and to make things worse, the water was splashing on all the computers. We frantically wrapped up the machines in towels and started collecting the rainwater, a process that took us a few hours till the rain finally ebbed. I was paranoid, thinking that the NUC was done for. I let it dry and took the GPU box and the NUC to a computer repair guy, who cleaned it out and said that luckily there was no sign of water damage.
After that scare, I finally got to work on my videos. I tried out a bunch of videos, and was pleased with some, and not so with others. Even though the system hold great promise, the videos I generate are simply too hard, with the combination of image blur and poor resolution just a poor fit for this sort of tracking. With this in mind, when we recorded the next bunch of videos, I took great care to minimise these sorts of problems and it worked great.