SuperSeeded optimises the commercial landscape market to reduce supply risk for growers, save time and earn money for the nature professions.

Get in touch

I invented a new way of counting visually patterned items with your eyes

I invented a new method of counting things with your eyes, "bisection counting" of periodic setructures. The idea is to use your eye to spot the midpoint of a linear/radial group of things (eg: fence palings, wheel spokes, or tiles), then keep halving the group again and again until you're down to just one (or nearest to one) item. Raise 2 to the power of halves you counted and you get a pretty fast and accurate estimate.

Calculate the Total:

Use the formula:

\[\text{Estimated Total} = 2^{\text{Number of Halvings}}\]

So if you halved 4 times:

\[\text{Estimated Total}=2^4=16\]

Error Bound estimation

• You visually estimate \(n\) halvings
• Your total estimate is \(2^n\)
• The actual count is \(N\)

The maximum relative error occurs between powers of two:

If \(N\) is between \(2^n\) and \(2^n+1\), and you guessed \(n\), then:

\[ \text{Relative error} = \frac{|2^n - N|}{N}\]

Which means the maximum error is just under 50% (specifically, ~41% at the midpoint between powers of two).

In practice, the actual error would be much smaller, because:

• Users rarely misjudge an entire halving level.
• The uncertainty is usually confined to the last halving, which only introduces small errors, not a full ±1 in \(N\).

So typical errors would be lower, around 5–15%

Soils - Elke Haege

Elke, together with soil scientist Simon Leake, SESL Australia (www.sesl.com.au) are about to launch an exciting publication through CSIRO titled: “Soils for Landscape development: Selection, Specification and Validation” .

Elke Haege Landscape Architecture + Consulting Arborist

This post is based on the work of Elke Haege Thorvaldson and Simon Leake (https://www.google.com/search?q=elkeh.com.au and SESL Australia). Elke Haege Thorvaldson is the owner of the calculator, and is an award-winning landscape architect and consulting arborist. The soil calculator, also known as the Soil Volume Simulator (SVS), was a collaborative effort between her and Simon Leake, a soil scientist from SESL Australia

Improving the calculator with continuous variables

This fantastic simulator generates soil volume requirements for trees based on their "designed size". The simulator includes surrounding soil, shared root zones, climatic conditions, maintenance, etc.

Strangely, discrete categories are used for things that have real measurements:

A number of interesting things can be done to improve it:

1. Increase precision by using continuous inputs instead of discrete ones
2. Find alternative ways to achieve required soil volumes by allowing some variables to be "driven" by others that are fixed and unchangeable.
3. Discover trade-offs among factors that can be combined to reduce time/cost to achieve required soil volume
4. Discover the key factors contributing to the volumes; what mattered the most?
5. Use symbolic regression to derive a simpler equation that could be easier to remember and apply from memory.
6. Develop a new interface that is more fluid and fun to use.

How I reverse engineered the simulator

The first task was to get tabular data from the calculator by inputting all possible combinations.

What didn't work:

• Browser automation to click all combinations of inputs using puppeteer to compile a csv
• Duplicating the Wordpress plugin and environment locally to examine its construction.
• Asking GPT-4 to produce an equation using all the factors

Apart from basic operators + - × that calculate answers to values input through drop-downs, there are sets of numbers conditional upon the initial Tree Design Size and Height input.

Since there are between 3 and 5 choices across 8 inputs multiplied by the four Tree Design Size and Height options available, the Cartesian product yields 23,040 combinations of inputs to get all possible soil volumes.

I felt that underneath this calculator, there were more interesting things to discover:

• Are there trade-offs between some factors that could shortlist an optimal set of candidates combinations?
• What were the main factors contributing to the volumes? what mattered the least?
• How to lock some factors and let others be driving?
• Could I train a machine learning model to predict soil volumes allowing any range of continuous variables unconstrained by the current options.

Examining the source code revealed 34 "elements" in an obscure data structure used by the Wordpress plugin.

Using LLMs to reverse engineer formulas is a novel idea which doesn't always work, but sometimes it did:

With some prompting and manual effort, I was able to intuit my way to several correct equations.

However, almost all inputs were mapped to discrete categorical variables in a way that further obscured the underlying equations

The values for Tree Design Size and Height are scaled away from empirical values (eg: small to 4m = 6.0; small/medium 4-9m high = 10.0)

In my symbolic regression analysis of the tree soil calculator, I discovered that while the original interface used discrete categories, the underlying math revealed continuous relationships between variables. Using PySR (Symbolic Regression), I identified that tree height is the dominant predictor of soil volume requirements, with a near-linear relationship (\(y₁ = 1.6387 * text{tree_height}\)) for total soil volume. For tree pit volume specifically, the algorithm uncovered a more complex formula incorporating soil suitability, maintenance factors, and planting replacement time, but still primarily driven by tree height. The percentage of soil volume outside the tree pit inversely affects the tree pit volume requirement, acting as a key modifying factor.

These equations demonstrate why the calculator could be improved by accepting continuous measurements rather than discrete categories. By extracting these mathematical relationships, I've created a more flexible model that allows for precise calculations with any input values, eliminating the constraints of the original categorical approach while preserving the essential relationships between variables.

Symbolic Regression with PySR

In 2023, CSIRO published Responsible AI: Best Practices for Creating Trustworthy AI Systems.

The book is a compilation of guidelines to assist in implementing governance, process and product considerations into AI systems:

"Responsible AI is the practice of developing and using AI systems in a way that provides benefits to individuals, groups, and wider society, while minimizing the risk of negative consequences." (Xu, Whittle, Zhu, & Lu, 2023, p. 18).

How can we make use of this fantastic resource to support development teams, consultants and governing bodies? I'm going to attempt to improve on efforts to operationalise research by the team at CSIRO Data61.[[2]]

Improving the usability of RAI patterns

We start with the notion that product, process and governance practices are causally interrelated and complex.

This departs from most conceptualisations of RAI patterns as containment hierarchies or trees:

The hierarchies above cannot adequately express complexity—instead we can model RAI practices as a directed graph of causal connections:

Why use a graph instead of organising content by theme?

1. Actions toward RAI can be interpreted based on their cause and effect
2. RAI patterns can be re-organised to support actions with the maximum impact.

By finding reinforcing feedback loops in our graph, we can prioritise actions that accelerate accountability, transparency, reliability and safety—or alternatively, balancing loops can limit or goal-seek to mitigate risk or harm, etc.

Inferring the causal structure of RAI

Thankfully the written compilation of patterns is already well structured[[3]], so with some data wrangling, it can be converted into a dataframe:

How can we convert this into a graph?

We can use a Large Language Model (LLM) and some procedural prompt engineering to follow constraints and conditions. Some common techniques were used to create an effective prompt:

• Chain-of-thought (COT) [[4]]
• EmotionPrompt [[5]]
• Expert prompting [[6]]
• Generated knowledge [[7]]
• Least to most [[8]]

Below is a snippet of the main function only — making use of the fantastic️ Python package LMQL and chat-gpt 3.5 turbo instruct for the LLM model.

Unsurprisingly the results are typical of all LLMs to date; impressive and imperfect. But this provides a useful starting point for the work of skilled modelers.

The graph below is generated from the chapter G.4. Regulatory Sandbox (Xu, Whittle, Zhu, & Lu, 2023, p. 65). In the graph, both balancing and reinforcing loops are present and although most loops seem plausible, the intermediate causal factors between nodes can be absent.

Regulatory Sandbox

Looking for leverage

Let's discover the feedback loops that may reveal actions or events that are mutually reinforcing, or balancing with “goal-seeking” behaviour.

Remember, these causal factors (dots) are present across different patterns, and so merging them uncovers even more feedback loops!

First a lovely hairball appears when aggregating all RAI factors

To reduce the number of nodes, we can merge them based on duplicates or similar meaning. In the scatter plot below, I used word vector embeddings to show causal factors across RAI patterns (dots) clustered by similar meaning/context. I use the PaCMAP dimensional reduction and cosine similarity with sentence-transformers/all-mpnet-base-v2 embeddings model.

Hover to see the corresponding text:

The next step is to find the feedback loops. We can use a graph algorithm to find them for us using networkx; the simple_cycles function:

If we merge based on these clusters and duplicate names, we can use graph algorithm reinforcing or balancing feedback loops.

42 loops were found

Let's examine some of the loops to see if they consist of actions or events that accelerate measures of accountability, transparency, reliability, safety– or limit risk, harm, etc.

The first one has 9 factors causally linked, where every factor increases or adds to the next one. This is a "B" balancing loop.

Let's observe a few important points about this graph

1. It's a loop because the path starts and finishes at the same factor; each one is causally linked to the next; a "directed cycle" in graph theory.
2. It's balancing because the count of red/blue polarities (a subjective judgement about its effect) is odd — Eg: an odd number of U-turns keeps you headed in the opposite direction.[[9]]
3. Balancing loops are "goal seeking" because they counteract deviations from a target state. Increasing any of these factors will continue to drive the system towards that state.

There are however are few problems with this loop:

1. Not all factors are described as sensible quantities. Better prompts may significantly improve LLM output and the resulting causal graph.
2. What is driving the loop? it helps to view factors outside the loop:

It would be exciting to uncover feedback loops that consist of factors from multiple RAI patterns. In the loop above, the factors that comprise it are described in a single RAI pattern; the "D.11. Digital Twin". Despite links between those factors and others outside of the Digital Twin pattern, none of them combined to form this particular loop. Finding these "hybrid factor" loops is an interesting direction for further research.

How reliable are these causal graphs?

The quality, coherence and external validity of the RAI patterns published by CSIRO Data61 provides a solid foundation for causal analysis, and I believe it is safe to assume there are causal relationships among them.

Systems modelling is an established field of research, and "causal loop diagramming" on which my approach is based has been used extensively in various industries; climate policy simulation and COVID-19 policy are recent examples.

The reliability of AI-assisted and computational causal modelling presented here hinges on the validity of the computational approach. I believe this remains in the hands of the experience of the modeller whose work could benefit from the increased scale and quantity of models enabled by LLMs and graph analysis.

How useful are they?

Insofar as results are valid, causal graph modelling is a useful direction for operationalising research in responsible AI.

For developers and governing bodies of responsible AI, identifying which actions will maximise their efforts by taking advantage of virtuous or viscous cycles should support decision making.

In an upcoming article I will document the development of an app that improves the usability of RAI patterns.

Remember— all models are wrong, but some are useful. Or as Picasso put it:

We all know that art is not truth. Art is a lie that makes us realize truth, at least the truth that is given us to understand. The artist must know the manner whereby to convince others of the truthfulness of his lies. — Pablo Picasso, 1923

Further research

Some directions for further research are

1. Building a causal graph RAI tool/webapp
2. Detecting archetypes in causal graphs (see below images)
3. Prompt-tuning for causal graphs
4. Causal graph-enabled RAG

Bibliography

Xu, X., Whittle, J., Zhu, L., & Lu, Q. (2023). Responsible AI: Best Practices for Creating Trustworthy AI Systems. Pearson Education.

[[1]]: Lee, S. U., Perera, H., Xia, B., Liu, Y., Lu, Q., Zhu, L., ... & Whittle, J. (2023). QB4AIRA: A Question Bank for AI Risk Assessment. arXiv preprint arXiv:2305.09300.

[[2]]: The team at Data61 CSIRO have published a bank of 293 questions for AI risk assessment: "QB4AIRA" [[1]] which aims to integrate with other AI risk assessment standards and frameworks. Although a Smart Risk Assessment Tool was developed based on the QB4AIRA, the challenge was to find relevant and stage-specific questions according to feedback from one of two testing groups (Ibid., Page 6.)

[[3]]: A similar has been published here— Lu, Q., Zhu, L., Xu, X., Whittle, J., Douglas, D., & Sanderson, C. (2022, May). Software engineering for responsible AI: An empirical study and operationalised patterns. In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (pp. 241-242).

[[4]]: Jason Wei, et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." arXiv preprint arXiv:2201.11903 (2022).

[[5]]: Li, C., Wang, J., Zhang, Y., Zhu, K., Hou, W., Lian, J., ... & Xie, X. (2023). Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760.

[[6]]: Xu, B., Yang, A., Lin, J., Wang, Q., Zhou, C., Zhang, Y., & Mao, Z. (2023). ExpertPrompting: Instructing Large Language Models to be Distinguished Experts. arXiv preprint arXiv:2305.14688.

[[7]]: Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. L., ... & Hajishirzi, H. (2021). Generated knowledge prompting for commonsense reasoning. arXiv preprint arXiv:2110.08387.

[[8]]: Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., ... & Chi, E. (2022). Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625

[[9]]: Tip, T. (2011). Guidelines for drawing causal loop diagrams. Systems Thinker, 22(1), 5-7. https://thesystemsthinker.com/wp-content/uploads/pdfs/220109pk.pdf

AI and Human Reasoning: Qualitative Research in the Age of Large Language Models

Bano, M., Zowghi, D., & Whittle, J. (2023). The AI Ethics Journal, 3(1).

After reading "AI and Human Reasoning" (Bano et al. 2023) I decided to replicate the study to investigate a critical omission in its experimental design.

The authors compare AI and Human responses when tasked with labelling text according to Schwartz’s theory for Human Values. The aim was to understand how the reasoning abilities of LLMs compare to human comprehension in the context of qualitative research.

Humans and LLMs only agreed about 25% of the time when classifying a dataset of 50 Amazon Alexa product reviews.

Numerical error

A key figure in the article contained an simple numerical error:

The row ChatGPT + Human2 incorrectly sums to 60. All other rows equal to the sample size of 50.

The original data from the experiment was published and below is my reproduction. Several off-by-one discrepancies have not been verified manually.

How significant is the miscount?—It appears to have contributed to the interpretation of results according to the authors' remarks

"While AIs show varied levels of agreement with humans, it is noteworthy that ChatGPT has a significant agreement with Human2, suggesting that certain AI models might align more closely with certain human perspectives."

Validity problem

The experiment doesn't control for the effect of different prompts on AI responses. This presents a problem of internal validity for the study because as the authors acknowledge, prompts are a critical factor that can significantly influence the analysis of Large Language Models (LLMs):

"The composition and specificity of prompts can guide the models’ analysis and processing of the task, thus affecting the results. A well-structured, clear, and contextually rich prompt helps the LLMs focus on the essential aspects of the task, reducing the likelihood of errors"

As we will see that approach is necessary but not sufficient for consistently good results.

This effect cannot be ignored because under real world conditions researchers would likely refine their prompts, just as with keyword search using wildcards *, AND/OR operations, spellings, etc.

What effect does varying prompts have?

My experiment measured the effect of changing the prompt (without changing model—gpt-3.5-turbo-1106) and comparing results as in the original article.

Prompt changes improved results by up to 1.8× ranked by agreement with Human 1 and Human 2.

The prompt changes (in order of effectiveness):

1. Provide examples of answers and Schwartz's Human Values
2. Zero-shot Chain-of-Thought prompt [1]
3. LLM selects best out of 3 multiple answers
4. JSON output.

(See roadmap.sh and learnprompting.org for overview of techniques)

Results

Simply by changing the prompt, AI-generated responses improved by 1.8× and 1.4× when classifying the "Main Value" of Schwartz Human Values to reach agreement with Human 1 and 2 respectively.

To learn which reviews were resistant to prompt change (vertical white blocks or gaps) the agree/disagree outcome across all prompts was plotted as a heat map:

To indicate the contribution of "reasoning" behind improvements in prompt changes, vector word embeddings were computed from various prompts and Human responses.

In the chart below every dot represents a paragraph explaining the reasoning supporting the classification. Those paragraphs are converted into 768 length dimensions and then reduced to two dimensions shown here:

There are three spatially distinct clusters including local variations in the semantics of reasons.

Texts from all sources (human and AI) are co-located in each cluster, but human responses (blue/red) are less overlapping than AI generated ones (purple/green). This is likely caused by different idiosyncratic phrasing by Human 1 and Human 2.

But it also suggests that the reasons given by AI-generated classifications are consistent (at least semantically) despite the vast difference between the prompts used to generate them.

These improvements and semantic relationships indicate that changes to prompts have an impact on the subjective judgement of AI-generated responses.

Researchers performing LLM assisted classification can be expected to optimise their prompts. Any comparison between AI-generated and human classification should take this into account.

In the next article, I will delve into more advanced prompt engineering that will enable modifiable outputs to move value bias between Human 1 or 2.

APPENDIX

Chain of Thought Prompts:

Vector embeddings are fascinating byproducts of neural networks–parameters that the network learns during training on a particular task.

They've been incredibly useful to me in various applications, from clustering real estate listings to categorising financial expenditures and even analysing baby names.

However, there's a catch

😱 Embeddings convey only intrinsic meaning.

When we transform data into vector embeddings, we receive high-dimensional output data in return, which we often reduce to three dimensions for visualization.

Yet, if we encode words like "banana" and "lemon" into an embedding space, we can't simply tweak their positions to emphasise qualities like "yellowness" or "sweetness", without adding additional context—we might envision "fruit" positioned somewhere in between.

So embeddings are not semantically versatile; they are not intended to encapsulate meaning unrelated to the data.

This limitation makes sense because embeddings are the effect of learning to minimise a loss function that is specific to their training data and prediction tasks—usually natural language processing.

Method for manipulating embeddings by recursive encode-reduce transformation

I've been working on a method to manipulate embeddings by transforming them in an iterative sequence of aggregation, weighting and dimensional reduction.

Here is a very complicated way of saying something very simple, let:

• \(E_i\) be the vector embedding for the (i)-th input.
• \(w_i\) be the weight (coefficient) assigned to the (i)-th embedding.
• \(R(\cdot)\) represent the dimensional reduction operation (e.g., PACMAP).
• \(\oplus\) denote the concatenation operation.
• \(\circ\) represent element-wise multiplication (weighting).

The process can be expressed as a sequence of operations:

Concatenation and Weighting:
\[X_1 = \sum_{i=1}^{n} w_i \cdot E_i\]
Where (X_1) is the weighted sum of the embeddings.

Dimensional Reduction:
\[X_2 = R(X_1)\]
Where \(X_2\) is the result of applying the dimensional reduction operation to \(X_1\).

Repeat Concatenation and Weighting:
\[X_3 = \sum_{i=1}^{n} w_i \cdot X_2\]
Where \(X_3\) is the weighted sum of the vectors obtained from the previous step.

Repeat Dimensional Reduction:
\[X_4 = R(X_3)\]
Where \(X_4\) is the result of applying the dimensional reduction operation to \(X_3\).

Validation

Thank you to Zenta, old friend.

Employer: Urban & Public

An interactive map of the Adelaide Hills wineries using geospatial and interactive 3D elements.

Images of various social and large bipartite network analyses of projects and botanical data.