Notes on Cytometry ∩ Deep Learning
I’m on a forced break from work (long story) in Munich and I wondered what I could do with myself. Luckily, I found a few locals who are interested in deep learning and that led me to a conversation on applying deep learning methods to cytometry. I’m publishing my notes from our 2h chat. My background in biology is pretty much non-existent so below represents what I eked out by asking questions. I wanted to understand enough of biology to get to the core of the problem and see if deep learning can help.
Cytometry
Cytometry is “analytics for biological cells”. After Wikipedia:
variables that can be measured by cytometric methods include cell size, cell count, cell morphology (shape and structure), cell cycle phase, DNA content, and the existence or absence of specific proteins on the cell surface or in the cytoplasm.
This is a foundational piece utilized in various fields of biology and medicine. People studying immunology in research setting rely on cytometry to understand biological processes at the cellular level. I hear that cytometry is also important in drug discovery both for modeling cellular processing and for testing drug responses.
The role of cytometry and, more broadly, “tissue analytics” in drug development and in health research is discussed in this overview by Bernd Bodenmiller. I found it to be readable even for an outsider (introduction and diagrams help!)[0].
In Munich we chatted about a specific kind of cytometry called Chip cytometry[1]. In this method, cells are in fixed position, and are being repeatedly stained with with antibodies. Each type of cell will attract a specific configuration of antibodies. By observing the light corresponding to each antibody you get cell-antibody signature that is used to infer the type of a cell. Practically, you get a greyscale image per antibody (or antibodies) and people use a wide range of image processing and image recognition tools to infer cell types.
The chip cytometry is particularly good at producing large number of high resolution images that can be compared against each other. In Germany, Zellkraftwerk is a hardware manufacturer for chip cytometry.
Deep learning
Currently used tools for analysis of cytometric images rely on the old-generation computer vision algorithms with custom feature engineering and embedding biology-specific knowledge into them.
The PhD students I have met wondered if deep learning could be used for cell segmentation. Image segmentation is a bread-and-butter problem for Deep Learning and the Deep Learning community continuously makes progress in this area.
The task we discussed is to take greyscale images (channels) of observed antibodies and output pixel classification corresponding to marking cells. The core problem is to combine signal across channels when making a judgement about the region of the tissue. Other information like the observed shapes should be incorporated too. This is exactly what CNNs and methods derived from them (e.g. U-nets) are good at.
The question of data size required for training models popped up and the recent progress in utilizing transfer learning comes to mind. My estimate is that to classify e.g. 6 cell types one would need just 300–400 examples of cell segmentation. Transfer learning is the single most useful technique that counters the prevailing narrative that deep learning requires huge amount of data.
Having said that, images in cytometry are unusual compared to the inputs of regular segmentation tasks. Normally, you have an RGB image as an input and you assign each pixel a class. The RGB acronym comes from the name of three channels: Red, Green and Blue. In cytometry, you have 30–50 channels so taking off-the-shelf model trained on RGB images won’t work most likely. My guess is that one would need to somehow adapt the transferred layers (e.g. tweak batch norm layers) or figure out some clever preprocessing steps. This is just a wild guess without looking at the actual data. Still, I wanted to flag it as I think this is an interesting modeling/deep learning problem.
Other challenge with 30–50 channels is that memory consumption on GPU will be higher compared to regular images. This might make training more difficult but I don’t think it’s a significant obstacle. It would just require some engineering to get around or make training slower due to smaller batch sizes.
I wondered if there are any fundamental reasons why cell segmentation might be difficult to solve with deep learning. We talked about this and an interesting issue of the “spatial spillover” came up.
An image comes handy for explaining this issue:
Spatial spillover happens when cells are packed tightly enough that some of the light reflected off a marker of one cell is observed in the region occupied by a neighboring cell. In our example, we have four cells of type 1 surrounding a cell of type 2. There’s enough light of type 1 marker in the area of the cell type 2 that we risk of misclassifying the center cell.
I learned that no existing software handles this well. Current solutions involve using HDR images and manual tuning of thresholds per each channel. This is labor-intensive, requires domain-specific knowledge and the obtained thresholds are dataset-specific. When a new experiment is conducted, one needs to start over again the process of tuning thresholds.
Looking at this as a computer vision problem, a few properties stand out:
- spatial spillover is a local problem within an image, i.e. features of the part of the image located 20 cells away are not relevant
- detecting the case of spatial spillover requires detecting subtle
intensity gradients - locally, you need to combine features (e.g. intensity gradients) together to get a sense whether we see an instance of spatial spillover or just signal of another cell; gradients will be reversed in these two cases
What I just described is exactly what Convolutional Neuronets are good at! In other words, deep learning’s walk in the park. Tellingly, the current methods of dealing with spatial spillover are an instance of manual feature engineering that’s been prevalent in computer vision before the deep learning revolution transformed the field.
Why now?
The chip cytometry that outputs high quality images with dozens of channels is a recent development. I wondered if there was some breakthrough in hardware that enabled it. What I was after is a collision of two big shifts in disjoint fields: hardware shift in cytometry and deep learning shift in computer vision. Search for such an event is propelled by my personal belief that when one major shift happens, lots of opportunity is created. When two shifts collide, a real magic is unlocked.
I learned that, in this case, the focus on chip cytometry and wealth of data it produces required more of a mindset shift than a hardware breakthrough. Hardware capabilities were there for quite some time but only with a recent firm arrival of machine learning into natural sciences, from biology to cosmology, people in these fields started to be open to the “big data” way of conducting their work.
In biology and medicine, we’re in the very early (and exciting) days of applying deep learning. Cytometry seems to be a promising entry point with a specific problem formulation and readily available machine learning models, constantly lifted by the general progress in deep learning methods.
I was excited to actually go and train a segmentation model but we’ve run into intellectual property issues around sharing datasets. If you’re reading this and have an access to a good dataset or wonder what would be a good dataset to collect, drop comment under this article or find me on Twitter.
[0] this was published in April 2016; I’m curious how authors’ views changed since then, especially in light of rapid progress of adapting deep learning in natural sciences
[1] the word chip has a different meaning in biomedical context than we understand it in computing world; there are no transistors involved here
