Welcome to the -viz suite of tools: classic bioinformatics tools, reimagined for the browser.
agviz · bamviz · genbankviz · phispyviz
Each of our tools is a classic bioinformatics tool, but reimagined for your browser. Your data stays where it belongs, on your computer!
This is post-cloud bioinformatics. Computers are powerful enough, even laptops, that you don’t need servers to do a lot of straightforward bioinformatics analysis, and the -viz tools enable you to do just that.
Try them out:
agviz – assembly graph visualisation for your browser
Recently, we’ve been vibe coding bioinformatics projects and vibe analysing bioinformatics data. Have we reached the singularity yet?
Vibe coding bioinformatics
We’ve been writing bioinformatics tools for over 20 years, but over the last year, vibe coding has completely changed what we do, and how we do it.
A typical bioinformatics tool that we’d write previously was predominantly written in Python (or back in the day, in Perl), and maybe had a bit of C or C++ to make things go faster.
Nowadays, with vibe coding, we are no longer limited to a single language. We can seamlessly integrate C++, Python, Node.js, Rust, TypeScript, JavaScript, Svelte, and a host of other languages.
For example, here are a few of our recent projects:
genbank viewer to visualise GenBank files in your browser. This project uses Rust, TypeScript, Svelte, and JavaScript
genbank_to converts genbank files from one format to another.
genome entropy to compare the entropies of DNA, amino acids, 3Di encodings, and 12-state encodings. This project uses Python, Jupyter Notebook, Shell, C, and Makefiles
oligo designer to permute and design random oligos and structured oligos. This project is primarily Python, but also uses JavaScript and HTML.
PhiSpy web for in-the-browser identification of prophages. This project uses TypeScript, CSS, JavaScript, and HTML, although of course PhiSpy is written in Python and C and is exposed via pyodide.
To build these projects, we use an agentic workflow, generally relying on codex to do most of the work for us.
Installation of codex is straightforward:
curl -fsSL https://chatgpt.com/codex/install.sh | sh
codex auth login
codex
Partly because we are working on a machine which automatically purges untouched documents after 21 days, and partly because it is generally best practice, especially for code, we always work in GitHub repository.
We created an AGENTS.md markdown file which resides in our ~/.codex/ directory, and puts some boundaries on codex, although this also authorises it to do a lot of work to the git repository.
For each project, we define the expected goals and required outcomes, and work in small steps. A common prompt that we use now is something like:
I have merged this PR. Please checkout main and update it, delete this local branch, and then identify the next milestone along the development pathway. Implement the steps to achieve that milestone, commit them, and make a PR. Once you have made the PR wait for the reviews to complete, and then pull the reviews and comments. Fix any outstanding comments, resolve the threads, and then push the updated changes back to the PR.
With this prompt, we are getting codex to basically do all the work. Of course, the key is making sure that the milestones are tightly defined, with appropriate test cases if possible, and that codex doesn’t wander down a rabbit hole. You need to keep an eye on that.
Vibe analysing
Based on our experience with vibe coding, we started vibe analysing data. We start with a similar strucutre, but because we are analysing data, we don’t often work in a git repository, because git can’t handle large datasets easily (there is a workaround with git-lfs but its not really suitable for metagenome scale data).
Nonetheless, the git structure provides a lot of advantages, especially when working with agentic code, so we create a local git repository using git init, and then we can add some of our code to the repository. Later, if we want, we can upload that to GitHub.
We also use an AGENTS.md markdown file to define what the agent should be doing, but here our target is not to make code, but to answer the scientific questions defined in SCIENTIFIC_BRIEF.md. This example is one we are using to analyse a phage genome. We combine those two with an ANALYSIS_PLAN.md, which translates the scientific brief into actionable items.
We provide blank DECISIONS.md and STATUS.md that the agent can append to as it develops its solutions to our problem.
Here is an overview of the primary files we have:
File
Question it answers
How Codex uses it
AGENTS.md
How must the agent behave while doing the work?
Supplies persistent operating rules: scientific validity first, source data read-only, explicit assumptions, durable evidence, and reporting obligations. Codex reads this automatically when working in the directory.
SCIENTIFIC_BRIEF.md
What are we trying to discover, and why?
Establishes hypotheses, scientific questions, available evidence, scope, non-goals, and success criteria.
ANALYSIS_PLAN.md
What evidence and analyses could answer those questions?
Why were consequential methodological choices made?
Preserves choices about thresholds, exclusions, models, statistical methods, and changes of direction.
STATUS.md
What is currently known, uncertain, blocked, or next?
Acts as concise scientific working memory across sessions. It should reflect evidence, not merely task completion.
README.md
How should a person adopt and operate this template?
Provides the human-facing entry point.
In addition, we use an immutable input/ directory with data that the agent should start with and should, under no circumstances, change. We have a mutable results/ directory where the agent can write new data, analyses, or outputs.
We keep several directories for additional outputs: report/ contains a markdown written report, and the figures directory contains images that are linked into the report, while notebooks contains a growing collection of Jupyter notebooks used to make those figures. We use scripts/ for additional code that the agent needs to analyse the data, and a work/ directory to give the agent a space to work in.
Overall, our directory structure looks something like this:
This is probably how you would end up with a directory if you were doing the analysis yourself, but I know from experience that you start with everything in one directory, and only when you accumulate enough cruft do you begin to move things around! With vibe analysing, it’s easy to accumulate so many files, and better to get some organisation going from the beginning.
I don’t think we’re at the singularity yet, but we are really, really close. I think we’re living in the transition period, where the singularity is upon us, and at some point—which we may not recognise—it will have passed us!
For vibe coding, we don’t write code any more. All of the code is written by AI. The user doesn’t need to know the language, or necessarily care about the language2 to write effective (but not necessarily efficient) code. We are in the singularity, but not through it; otherwise the AI would know the next questions and milestones and be already coding towards them before I’ve even realised that we are done with the last milestone.
For vibe analysis, we are also close. With a single prompt, I can get AI to go search the SRA, use fasterq-dump to download the sequences (based, presumably, on my blog posts), and compare them to genomes of interest. All this is hands-off, and the AI runs the analysis. But it’s not there designing the analysis yet. It doesn’t ask the interesting questions.
However, AI is helping with the interesting questions. It is providing quicker ways to get to the interesting questions, and helping me filter out the less interesting questions. It’s downloading the data, processing the data, and letting me spend time thinking about the results. We are not AI, but we are definitely in the midst of Intelligence Amplification—IA.
Have we passed the singularity? No, not yet. Is it close? Definitely, it feels like the last year or two it has got so much closer.
Vernor was also a Professor at San Diego State University, and when I was recruited there as a lowly Assistant Professor, I inherited Vernor’s office. Alas, his brilliant writing was not, apparently, caused by the office. ↩︎
I believe this is true, but it is different from not knowing or caring about computer science fundamentals. For example, I need to remind AI not to use an O(n2) algorithm from time to time. ↩︎
This blog post wasn’t written by AI, but the featured image was created by AI. ↩︎
Note: This is part of our series on Pawsey that are written by users – not by Pawsey staff. There are certainly other, and probably better, ways to do this, but this is what we are currently doing! You should also read the Pawsey Help Documentation
Pawsey storage locations (disks)
There are three main storage locations that you can access:
/home (where you log in to) has a limit of 10,000 files and 1Gb of storage, so you will quickly fill that up.
/software has a limit of 16,384G abd 250k files, so you can put more things there, but not everything
/scratch has 9.8P of storage, but everything is deleted after 21 days, so this is not a brilliant location either.
acacia is for longer term storage but you can’t access that directly, so you can’t install software there.
A basic conda set up.
I use /software for some basic conda environments that I am going to use regularly. For example, I have an rclone environment that only has rclone and I use to move data on and off of setonix or acacia. My other environments are a bioinformatics environment which has a few common tools I use day-to-day like samtools and minimap and a git-lfs environment I also use regularly, that only has git-lfs installed. (If you don’t know what git-lfs is for, you probably don’t need it!)
Everything else, I put in a temporary directory in /scratch and then I recreate them as I need it.
There are two different solutions to this problem, and I use both depending on how I feel.
Disposable /scratch conda environments.
I make a temporary environment on /scratch with a directory name that is a meaningless random set of characters. I install what I need, use it as I need it, and then later, when I remember, I delete the environment.
The advantage of this approach, is you leave it if something is broken and start again, and you make a new directory for each thing you are doing.
Rememberable, but disposable, /scratch conda environments.
The alternative is to use a name that you will remember, but then you also need to remember that things are probably broken after 21 days and you need to reinstall everything.
Let’s walk through setting up your conda, installing some software, remembering how to do it, and deleting the environment.
For this example, I’m going to use autocycler as my software to install, and I’ll also install minimap2 and samtools.
Go to the instructions for installing miniforge on a Unix-like platform and use either wget or curl to download the installer. It doesn’t matter which one, so start with curl (because that is first on the list), and if that doesn’t work use wget.
Set up your .condarc file.
Use nano ~/.condarc and copy the block below and paste into the file.
This block adds conda-forge and bioconda so you can easily install software, sets the default environment directory to /software and the location where the files are downloaded to /scratch.
Create a environment file to install the software
If you use environment files, you can install the software directly from the file, and then if you need to reinstall things (e.g. because the file has been deleted, you just need one command!).
Use nano to create a file called environment.yml and paste this information:
[NOTE!] The environment created without using a --prefix command is called autocycler and is on /software. The environment created with the --prefix command is on /scratch and is a different environment. Since this is exceptionally confusing, do one or the other, but NOT both!
Deleting environments
Pawsey will automatically delete any files that are older than 21 days, so you don’t need to worry about old environments, however it gets very confusing, so you should delete them.
Start by doing mamba info --envs to get a list of your environments, and then choose the path of the one you want to remove.
Delete the environment, and any files left in it, using
Sometimes when you are installing software you will get random errors about packages being incomplete or not able to be installed. Usually, the problem is that the packages on /scratch have been deleted, so clean them out and try again, which will force them to be re-downloaded.
In metagenomics, we often face a familiar trade-off: do we use a smaller, more clustered reference database that runs quickly, or a larger, more detailed database that may give finer resolution but requires substantially more compute?
One common example is the choice between the clustered databases from UniRef. We use both UniRef50 and UniRef100 for protein-based taxonomic or functional classification. Both databases derive from UniProt, but they differ in clustering level. UniRef50 clusters sequences at 50% identity, producing a smaller, less redundant database. UniRef100 retains much more sequence-level detail and is therefore larger, more comprehensive, and more computationally demanding. At the moment, the mmseqs2 version of UniRef100 is 275G while the mmseqs2 version of UniRef50 is 29G.
The obvious assumption is that UniRef100 should be “better” because it contains more information. But in metagenomics, the more useful question is often: does UniRef100 change the biological interpretation enough to justify the additional runtime?
In one recent comparison using the same metagenomic dataset, the UniRef50 search took 503 seconds using 64 threads. The equivalent UniRef100 search took 9,846 seconds using the same 64 threads.
That is approximately:
8 minutes 23 seconds for UniRef50
2 hours 44 minutes 6 seconds for UniRef100
an almost 20-fold increase in runtime for UniRef100 (even though its ~10x the size)
That is not a minor computational penalty. On a single dataset, it may be acceptable. Across hundreds or thousands of metagenomes, it becomes a major consideration for throughput, queue time, storage, and reproducibility.
What changed in the results?
At both genus and family taxonomic levels, the UniRef50 and UniRef100-derived profiles were broadly concordant, especially after log transformation. This is important because metagenomic abundance tables are typically sparse and highly skewed: a small number of taxa or functions can dominate the raw counts, while most features are rare or absent in most samples.
At the family level, the agreement was especially strong. Comparing family-by-sample abundance values, the raw Pearson correlation was modest, but this was expected because raw count data are sparse and unevenly distributed. After applying a log1p transformation (y=log(1 + x)), the correlation was much stronger. Family-level total abundances across the dataset were also well correlated.
The sample-level totals were almost perfectly correlated between the two database choices: the two approaches largely agreed on which samples had higher or lower overall assignment levels.
However, the two approaches were not identical. UniRef100 assigned more total counts and, in some cases, appeared to resolve reads that were left at broader or more ambiguous taxonomic levels in the UniRef50-derived result.
The practical interpretation
The key point is not that UniRef50 and UniRef100 are interchangeable. They are not. UniRef100 can produce more assignments and may give more specific taxonomic placements in some parts of the tree.
But for many metagenomic questions, especially when working at higher taxonomic ranks such as family, the broad biological signal may be very similar between UniRef50 and UniRef100. If the main goal is to compare samples, identify large-scale community shifts, or screen many datasets efficiently, UniRef50 may be more than adequate.
If the goal is to chase fine-scale taxonomic differences, resolve difficult clades, or maximise the number of assigned reads, UniRef100 may be worth the cost.
A useful rule of thumb
For large-scale metagenomic projects, I would treat UniRef50 as the sensible default for exploratory analysis, cohort-scale comparisons, and routine pipelines. It is faster, cheaper, and often preserves the major biological patterns.
UniRef100 is better reserved for cases where the additional resolution is likely to matter: detailed reanalysis of selected samples, validation of specific signals, fine-grained taxonomic interpretation, or situations where ambiguous assignments from UniRef50 need to be resolved.
A practical workflow might be:
Run the full dataset against UniRef50.
Identify the major biological patterns, outliers, and features of interest.
Re-run selected samples or specific analyses against UniRef100.
Ask whether UniRef100 changes the conclusion, not just whether it changes the counts.
This gives the best of both worlds: speed and scalability from UniRef50, with targeted use of UniRef100 where the extra resolution may be informative.
Bigger databases are not automatically better
Metagenomics already has enough computational bottlenecks. Bigger databases increase runtime, memory pressure, disk usage, and downstream complexity. They can also increase the number of plausible matches, which does not always make interpretation easier.
The important question is not simply, “Which database is larger?” or “Which database is more complete?”
The better question is:
Does the larger database change the biological conclusion enough to justify the additional computational cost?
In my comparison, UniRef100 required nearly 20 times longer than UniRef50. The resulting profiles were not identical, but the major sample-level and family-level abundance patterns were strongly concordant.
Install the usual suspect python packages that you will probably want to use. For convenience, I usually put these in a file in my Git repo called requirements.txt.
When you open jupyter there is a list of kernels that you can connect to. (If you have a window open that list will be on the top right.) If you rename your jupyter kernel it makes it much easier to find the kernel associated with this conda environment. The default name is something like Python 3 which is not helpful if you have lots of them!
a. Find where your kernel is installed
This command shows your jupyter kernels
jupyter kernelspec list
You’ll see your kernel(s) and the locations of them. In the location listed there is a file called kernel.json.
b. Edit that file:
vi $HOME/miniconda3/envs/gpu_notebook/share/jupyter/kernels/python3/kernel.json
c. Change the name to be meaningful
Change the value associated with the display_name key. Set it to something meaningful so you can find it in your browser!
4. Set up the XLA_FLAGS environment variable.
This was essential for me to get tensorflow working. There is a directory somewhere in your conda environment with the libdevice library that is needed. For my installation that was in nvvm/libdevice/libdevice.10.bc. Of course you can find yours with:
find ~/miniconda3/ -name libdevice
You want to set the XLA_FLAGS variable to point to the base of the nvvm folder. This command sets it inside the conda environment so it is always set when the conda environment is activated, and unset when it is deactivated.
conda env config vars set XLA_FLAGS=--xla_gpu_cuda_data_dir=$HOME/miniconda3/envs/gpu_notebook
5. Activate the environment
Don’t forget to submit this to a node with GPU capabilities!
When building linear mixed models with Python’s statsmodules module, I repeatedly, and often incoherently, ran into np.linalg.LinAlgError errors that are Singular matrix errors.
There are a couple of things to check for with these errors:
First, drop any rows where there are NaN values for the predictors:
e.g. if your predictors are in a list called predictors, try this
Finally, when all that doesn’t work, you should try different methods to fit the model. These are the methods I currently use, and I try them in this order and save the results for the first one that completes.
results = None
for meth in 'bfgs', 'lbfgs', 'cg', 'powell', 'nm':
try:
result = model.fit(method=meth)
print(f"Method {meth} PASSED", file=sys.stderr)
break
except np.linalg.LinAlgError as e:
print(f"Method {meth} failed", file=sys.stderr)
if results:
print(results.summary)
I needed an updated node.js on my WSL … for some reason ubuntu 20 has quite an old version.
First, as sudo, I removed all the old versions to avoid any conflicts:
apt purge nodejs
apt auto-remove
The purge command completely removes it, while the autoremove removes any dependencies. That should be optional, because those should work once we install the new node.js versions, but YMMV.
Next, install the node.js installer, called nvm for node version manager
To begin, set up your install location. For a single install it should be in ~/.nvm but for a site-wide install, you might want to install nvm in /usr/local/nvm.
Put these lines in your ~/.bashrc
export NVM_ROOT_DIR=$HOME/.nvm
# export NVM_ROOT_DIR=/usr/local/nvm ## << use this version if you are doing a site wide install
export NVM_DIR=$HOME/.nvm
[ -s "$NVM_ROOT_DIR/nvm.sh" ] && . "$NVM_ROOT_DIR/nvm.sh"
[ -s "$NVM_ROOT_DIR/bash_completion" ] && . "$NVM_ROOT_DIR/bash_completion"
NVM_BIN=$HOME/.nvm/versions/node/v23.3.0/bin ## << you may need to change this line
NVM_INC=$HOME/.nvm/versions/node/v23.3.0/include/node ## << you may need to change this line
export PATH=$NVM_DIR:$NVM_BIN:$PATH
wsl -bash: node: cannot execute binary file: Exec format error
then try installing a different version
node install --lts
For me, using the default node install node didn’t work because of the binary file Exec format error, while using node install --lts worked like a charm!
