Validate & register flow cytometry data

Flow cytometry is a technique used to analyze and sort cells or particles based on their physical and chemical characteristics as they flow in a fluid stream through a laser beam.

Here, we’ll transform, validate and register two flow cytometry datasets (Alpert19 and FlowIO sample) to demonstrate how to create and query a custom flow cytometry registry.

!lamin init --storage ./test-flow --schema bionty

Show code cell output Hide code cell output

💡 creating schemas: core==0.47.5 bionty==0.30.3 
✅ saved: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-09-05 09:41:55)
✅ saved: Storage(id='9jegyU3p', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-flow', type='local', updated_at=2023-09-05 09:41:55, created_by_id='DzTjkKse')

✅ loaded instance: testuser1/test-flow
💡 did not register local instance on hub (if you want, call `lamin register`)

import lamindb as ln
import lnschema_bionty as lb
import readfcs

lb.settings.species = "human"

✅ loaded instance: testuser1/test-flow (lamindb 0.52.2)

ln.track()

💡 notebook imports: lamindb==0.52.2 lnschema_bionty==0.30.3 readfcs==1.1.6

✅ saved: Transform(id='OWuTtS4SAponz8', name='Validate & register flow cytometry data', short_name='facs', version='0', type=notebook, updated_at=2023-09-05 09:41:57, created_by_id='DzTjkKse')

✅ saved: Run(id='iYTDmcBCeX2NKHxXjGFh', run_at=2023-09-05 09:41:57, transform_id='OWuTtS4SAponz8', created_by_id='DzTjkKse')

Alpert19

Transform

(Here we skip steps of data transformations, which often includes filtering, normalizing, or formatting data.)

We start with a flow cytometry file from Alpert19:

ln.dev.datasets.file_fcs_alpert19(
    populate_registries=True,  # pre-populate registries to simulate an used instance
)

PosixPath('Alpert19.fcs')

Use readfcs to read the fcs file into memory:

adata = readfcs.read("Alpert19.fcs")
adata

AnnData object with n_obs × n_vars = 166537 × 40
    var: 'n', 'channel', 'marker', '$PnB', '$PnE', '$PnR'
    uns: 'meta'

Validate

First, let’s validate the features in .var.

We’ll use the CellMarker reference to link features:

lb.CellMarker.validate(adata.var.index, "name");

✅ 27 terms (67.50%) are validated for name

❗ 13 terms (32.50%) are not validated for name: Time, Cell_length, Dead, (Ba138)Dd, Bead, CD19, CD4, IgD, CD11b, CD14, CCR6, CCR7, PD-1

We see that many features aren’t validated. Let’s standardize the identifiers first to get rid of synonyms:

adata.var.index = lb.CellMarker.standardize(adata.var.index)

💡 standardized 35/40 terms

Great, now we can validate our markers once more:

validated = lb.CellMarker.validate(adata.var.index, "name")

✅ 35 terms (87.50%) are validated for name

❗ 5 terms (12.50%) are not validated for name: Time, Cell_length, Dead, (Ba138)Dd, Bead

Things look much better, but we still have 5 CellMaker records that seem more like metadata. Hence, let’s curate the AnnData object a bit more.

Let’s move metadata (non-validated cell markers) into adata.obs:

adata.obs = adata[:, ~validated].to_df()
adata = adata[:, validated].copy()

Now we have a clean panel of 35 cell markers:

lb.CellMarker.validate(adata.var.index, "name");

✅ 35 terms (100.00%) are validated for name

Next, let’s register the metadata features we moved to .obs:

# Feature.from_df creates feature records with type auto-populated
features = ln.Feature.from_df(adata.obs)

ln.add(features)

In addition, We’d also like to link this file with external features:

ln.Feature.validate("assay", "name")
lb.ExperimentalFactor.validate("FACS", "name");

✅ 1 term (100.00%) is validated for name

❗ 1 term (100.00%) is not validated for name: FACS

Since we never validated the term “FACS”, let’s search for it’s ontology and register it:

lb.ExperimentalFactor.bionty().search("FACS").head(2)

	ontology_id	definition	synonyms	parents	molecule	instrument	measurement	__ratio__
name
fluorescence-activated cell sorting	EFO:0009108	A Flow Cytometry Assay That Provides A Method ...	FACS|FAC sorting	[]	None	None	None	100.0
FACS-seq	EFO:0008735	Fluorescence-Activated Cell Sorting And Deep S...	None	[EFO:0001457]	RNA assay	None	None	90.0

facs = lb.ExperimentalFactor.from_bionty(ontology_id="EFO:0009108")
facs.save()

✅ created 1 ExperimentalFactor record from Bionty matching ontology_id: 'EFO:0009108'

Register

file = ln.File.from_anndata(adata, description="Alpert19", field=lb.CellMarker.name)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/rr4l4mthbDDiqvGZVTQa.h5ad')

💡 parsing feature names of X stored in slot 'var'

✅    35 terms (100.00%) are validated for name

✅    linked: FeatureSet(id='xzcrzZnuRxRmbysGYlTo', n=35, type='number', registry='bionty.CellMarker', hash='ldY9_GmptHLCcT7Nrpgo', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    5 terms (100.00%) are validated for name

✅    linked: FeatureSet(id='gZ37AL6OHH2Ryi0CAw1u', n=5, registry='core.Feature', hash='XIvdBiotSwMr9YUFfOoD', modality_id='psEzQ4pR', created_by_id='DzTjkKse')

file.save()

✅ saved 2 feature sets for slots: 'var','obs'

✅ storing file 'rr4l4mthbDDiqvGZVTQa' at '.lamindb/rr4l4mthbDDiqvGZVTQa.h5ad'

features = ln.Feature.lookup()
file.add_labels(facs, features.assay)
file.add_labels(lb.settings.species, features.species)

✅ linked new feature 'assay' together with new feature set FeatureSet(id='gaRmVU3yAXIkyY1teeiI', n=1, registry='core.Feature', hash='vw5vW9VQnL3DtMCqe0vN', updated_at=2023-09-05 09:42:06, modality_id='psEzQ4pR', created_by_id='DzTjkKse')

💡 no file links to it anymore, deleting feature set FeatureSet(id='gaRmVU3yAXIkyY1teeiI', n=1, registry='core.Feature', hash='vw5vW9VQnL3DtMCqe0vN', updated_at=2023-09-05 09:42:06, modality_id='psEzQ4pR', created_by_id='DzTjkKse')

✅ linked new feature 'species' together with new feature set FeatureSet(id='xwtWGUwhRD5WhVC4XT4Y', n=2, registry='core.Feature', hash='TxMV2SUjPylWjWQJtLQG', updated_at=2023-09-05 09:42:06, modality_id='psEzQ4pR', created_by_id='DzTjkKse')

file.features

Features:
  var: FeatureSet(id='xzcrzZnuRxRmbysGYlTo', n=35, type='number', registry='bionty.CellMarker', hash='ldY9_GmptHLCcT7Nrpgo', updated_at=2023-09-05 09:42:06, created_by_id='DzTjkKse')
    'CD123', 'CD86', 'CD45RA', 'CD28', 'CD27', 'CD85j', 'CD16', 'DNA1', 'Cd14', 'Ccr6'
  obs: FeatureSet(id='gZ37AL6OHH2Ryi0CAw1u', n=5, registry='core.Feature', hash='XIvdBiotSwMr9YUFfOoD', updated_at=2023-09-05 09:42:06, modality_id='psEzQ4pR', created_by_id='DzTjkKse')
    (Ba138)Dd (number)
    Dead (number)
    Cell_length (number)
    Time (number)
    Bead (number)
  external: FeatureSet(id='xwtWGUwhRD5WhVC4XT4Y', n=2, registry='core.Feature', hash='TxMV2SUjPylWjWQJtLQG', updated_at=2023-09-05 09:42:06, modality_id='psEzQ4pR', created_by_id='DzTjkKse')
    🔗 species (1, bionty.Species): 'human'
    🔗 assay (1, bionty.ExperimentalFactor): 'fluorescence-activated cell sorting'

Check a few validated cell markers in .var:

file.features["var"].df().head(10)

	name	synonyms	gene_symbol	ncbi_gene_id	uniprotkb_id	species_id	bionty_source_id	updated_at	created_by_id
id
n40112OuX7Cq	CD123		IL3RA	3563	P26951	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
L0m6f7FPiDeg	CD86		CD86	942	A8K632	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
a624IeIqbchl	CD45RA		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
CLFUvJpioHoA	CD28		CD28	940	B4E0L1	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
uThe3c0V3d4i	CD27		CD27	939	P26842	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
lRZYuH929QDw	CD85j		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
bspnQ0igku6c	CD16		FCGR3A	2215	O75015	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
YA5Ezh6SAy10	DNA1		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
roEbL8zuLC5k	Cd14		CD14	4695	O43678	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
fpPkjlGv15C9	Ccr6		CCR6	1235	P51684	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse

FlowIO sample

Let’s transform, validate and register another flow file:

Transform

There are no further transformations necessary.

adata2 = readfcs.read(ln.dev.datasets.file_fcs())

Validate

We’d like to track all features in .var, so we register them:

adata2.var.index = lb.CellMarker.bionty().standardize(adata2.var.index)

💡 standardized 14/16 terms

markers = lb.CellMarker.from_values(adata2.var.index, "name")
ln.save(markers)

✅ loaded 10 CellMarker records matching name: 'CD3', 'CD28', 'CD8', 'Cd4', 'CD57', 'Cd14', 'Cd19', 'CD27', 'Ccr7', 'CD127'

✅ created 4 CellMarker records from Bionty matching name: 'CCR5', 'CD45RO', 'Ki67', 'SSC-A'

❗ did not create CellMarker records for 2 non-validated FieldAttr(CellMarker.name)s: 'FSC-A', 'FSC-H'

Standardize synonyms so that all features pass validation:

adata2.var.index = lb.CellMarker.standardize(adata2.var.index)

💡 standardized 14/16 terms

lb.CellMarker.validate(adata2.var.index, "name");

✅ 14 terms (87.50%) are validated for name

❗ 2 terms (12.50%) are not validated for name: FSC-A, FSC-H

Register

file2 = ln.File.from_anndata(
    adata2, description="My fcs file", field=lb.CellMarker.name
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/HoPppwkIBbvggxZO2V9Y.h5ad')

💡 parsing feature names of X stored in slot 'var'

✅    14 terms (87.50%) are validated for name

❗    2 terms (12.50%) are not validated for name: FSC-A, FSC-H

✅    linked: FeatureSet(id='0CD8lXO9Tn0ssI3fZwYz', n=14, type='number', registry='bionty.CellMarker', hash='npy5P7AYbjKLInpXlNvb', created_by_id='DzTjkKse')

file2.save()

✅ saved 1 feature set for slot: 'var'

✅ storing file 'HoPppwkIBbvggxZO2V9Y' at '.lamindb/HoPppwkIBbvggxZO2V9Y.h5ad'

file2.add_labels(facs, features.assay)
file2.add_labels(lb.settings.species, features.species)

✅ linked new feature 'assay' together with new feature set FeatureSet(id='vgeSXw5KqcUrcMkT8JXN', n=1, registry='core.Feature', hash='vw5vW9VQnL3DtMCqe0vN', updated_at=2023-09-05 09:42:11, modality_id='psEzQ4pR', created_by_id='DzTjkKse')

✅ loaded: FeatureSet(id='xwtWGUwhRD5WhVC4XT4Y', n=2, registry='core.Feature', hash='TxMV2SUjPylWjWQJtLQG', updated_at=2023-09-05 09:42:06, modality_id='psEzQ4pR', created_by_id='DzTjkKse')

✅ linked new feature 'species' together with new feature set FeatureSet(id='xwtWGUwhRD5WhVC4XT4Y', n=2, registry='core.Feature', hash='TxMV2SUjPylWjWQJtLQG', updated_at=2023-09-05 09:42:11, modality_id='psEzQ4pR', created_by_id='DzTjkKse')

file2.features

Features:
  var: FeatureSet(id='0CD8lXO9Tn0ssI3fZwYz', n=14, type='number', registry='bionty.CellMarker', hash='npy5P7AYbjKLInpXlNvb', updated_at=2023-09-05 09:42:11, created_by_id='DzTjkKse')
    'Cd14', 'CD3', 'CD8', 'CD28', 'CD45RO', 'CD57', 'CD127', 'CD27', 'Cd19', 'SSC-A'
  external: FeatureSet(id='xwtWGUwhRD5WhVC4XT4Y', n=2, registry='core.Feature', hash='TxMV2SUjPylWjWQJtLQG', updated_at=2023-09-05 09:42:11, modality_id='psEzQ4pR', created_by_id='DzTjkKse')
    🔗 species (1, bionty.Species): 'human'
    🔗 assay (1, bionty.ExperimentalFactor): 'fluorescence-activated cell sorting'

file2.view_flow()

Query by cell markers

Which datasets have CD14 in the flow panel:

cell_markers = lb.CellMarker.lookup()

cell_markers.cd14

CellMarker(id='roEbL8zuLC5k', name='Cd14', synonyms='', gene_symbol='CD14', ncbi_gene_id='4695', uniprotkb_id='O43678', updated_at=2023-09-05 09:42:02, species_id='uHJU', bionty_source_id='ovpv', created_by_id='DzTjkKse')

panels_with_cd14 = ln.FeatureSet.filter(cell_markers=cell_markers.cd14).all()

ln.File.filter(feature_sets__in=panels_with_cd14).df()

	storage_id	key	suffix	accessor	description	version	size	hash	hash_type	transform_id	run_id	initial_version_id	updated_at	created_by_id
id
HoPppwkIBbvggxZO2V9Y	9jegyU3p	None	.h5ad	AnnData	My fcs file	None	6876232	Cf4Fhfw_RDMtKd5amM6Gtw	md5	OWuTtS4SAponz8	iYTDmcBCeX2NKHxXjGFh	None	2023-09-05 09:42:11	DzTjkKse
rr4l4mthbDDiqvGZVTQa	9jegyU3p	None	.h5ad	AnnData	Alpert19	None	33367624	14w5ElNsR_MqdiJtvnS1aw	md5	OWuTtS4SAponz8	iYTDmcBCeX2NKHxXjGFh	None	2023-09-05 09:42:06	DzTjkKse

Shared cell markers between two files:

files = ln.File.filter(feature_sets__in=panels_with_cd14, species__name="human").list()
file1, file2 = files[0], files[1]

file1_markers = file1.features["var"]
file2_markers = file2.features["var"]

shared_markers = file1_markers & file2_markers
shared_markers.list("name")

['Cd14', 'CD3', 'CD8', 'CD28', 'CD57', 'CD127', 'CD27', 'Cd19', 'Ccr7', 'Cd4']

Flow marker registry

Check out your CellMarker registry:

lb.CellMarker.filter().df()

	name	synonyms	gene_symbol	ncbi_gene_id	uniprotkb_id	species_id	bionty_source_id	updated_at	created_by_id
id
n40112OuX7Cq	CD123		IL3RA	3563	P26951	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
L0m6f7FPiDeg	CD86		CD86	942	A8K632	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
a624IeIqbchl	CD45RA		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
CLFUvJpioHoA	CD28		CD28	940	B4E0L1	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
uThe3c0V3d4i	CD27		CD27	939	P26842	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
lRZYuH929QDw	CD85j		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
bspnQ0igku6c	CD16		FCGR3A	2215	O75015	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
YA5Ezh6SAy10	DNA1		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
roEbL8zuLC5k	Cd14		CD14	4695	O43678	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
fpPkjlGv15C9	Ccr6		CCR6	1235	P51684	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
gEfe8qTsIHl0	CD24		CD24	100133941	B6EC88	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
0qCmUijBeByY	CD94		KLRD1	3824	Q13241	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
Nb2sscq9cBcB	CD57		B3GAT1	27087	Q9P2W7	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
hVNEgxlcDV10	CD127		IL7R	3575	P16871	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
4uiPHmCPV5i1	CXCR5		CXCR5	643	A0N0R2	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
0vAls2cmLKWq	ICOS		ICOS	29851	Q53QY6	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
0evamYEdmaoY	Igd		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
sYcK7uoWCtco	Ccr7		CCR7	1236	P32248	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
agQD0dEzuoNA	CXCR3		CXCR3	2833	P49682	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
L0WKZ3fufq0J	CD11c		ITGAX	3687	P20702	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
ttBc0Fs01sYk	CD8		CD8A	925	P01732	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
cFJEI6e6wml3	CD20		MS4A1	931	A0A024R507	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
h4rkCALR5WfU	CD56		NCAM1	4684	P13591	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
8OhpfB7wwV32	Cd19		CD19	930	P15391	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
N2F6Qv9CxJch	CD11B		ITGAM	3684	P11215	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
CR7DAHxybgyi	CD38		CD38	952	B4E006	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
HEK41hvaIazP	Cd4		CD4	920	B4DT49	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
k0zGbSgZEX3q	HLADR	HLA‐DR|HLA-DR|HLA DR	None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
50v4SaR2m5zQ	CD25		IL2RA	3559	P01589	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
yCyTIVxZkIUz	DNA2		DNA2	1763	P51530	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
a4hvNp34IYP0	CD3		None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
c3dZKHFOdllB	CD33		CD33	945	P20138	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
4EojtgN0CjBH	CD161		KLRB1	3820	Q12918	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
ljp5UfCF9HCi	TCRgd	TCRGAMMADELTA|TCRγδ	None	None	None	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
2VeZenLi2dj5	PD1	PID1|PD-1|PD 1	PDCD1	5133	A0A0M3M0G7	uHJU	ovpv	2023-09-05 09:42:02	DzTjkKse
XvpJ6oL3SG7w	CD45RO		None	None	None	uHJU	ovpv	2023-09-05 09:42:10	DzTjkKse
VZBURNy04vBi	SSC-A	SSC A|SSCA	None	None	None	uHJU	ovpv	2023-09-05 09:42:10	DzTjkKse
Qa4ozz9tyesQ	Ki67	Ki-67|KI 67	None	None	None	uHJU	ovpv	2023-09-05 09:42:10	DzTjkKse
UMsp5g0fgMwY	CCR5		CCR5	1234	P51681	uHJU	ovpv	2023-09-05 09:42:10	DzTjkKse

Show code cell content Hide code cell content

# a few tests
assert set(shared_markers.list("name")) == set(
    [
        "Ccr7",
        "CD3",
        "Cd14",
        "Cd19",
        "CD127",
        "CD27",
        "CD28",
        "CD8",
        "Cd4",
        "CD57",
    ]
)
ln.File.filter(feature_sets__in=panels_with_cd14).exists()

True

Show code cell content Hide code cell content

# clean up test instance
!lamin delete --force test-flow
!rm -r test-flow

💡 deleting instance testuser1/test-flow
✅     deleted instance settings file: /home/runner/.lamin/instance--testuser1--test-flow.env

✅     instance cache deleted

✅     deleted '.lndb' sqlite file
❗     consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/test-flow