How I Actually Know If My Scripts Are Useful
A few months ago, I finished a Grasshopper script for a project. It automated an annoying clash-checking process that used to take an engineer about 4 hours, now done in 20 minutes with the help of the script.
By any measure, that’s a win.
But six months later, I couldn’t tell you if anyone was still using it. I didn’t know if it had been adapted for other projects, if it was sitting in a folder somewhere or if anyone even remembered it.
I had no way to know if my hard work was actually meaningful at the end of the day.
When I hand over a solution to someone, all I get is their thanks and sometimes they call me if they have a problem. But on a normal day, I have no clue.
A big motivation for me with work is knowing how impactful it is to people. I would happily work weekends and nights if I knew that my hard work was meaningful. It’s one of the main reasons I preach about understanding before solving anything because I want to work on things that are useful.
But in computational design (CoDe), measuring that impact is genuinely hard. Some solutions truly help people, like automating a 2-hour manual data entry task down to 5 minutes. But sometimes we also build things no one asked for. We just don’t know.
This is part 2 of a two part series in exploring what it means to measure the value of CoDe. In Part 1, we explored why it’s so difficult, what the common metrics are: time saved, risk reduced, quality improved and why each one alone doesn’t tell the full story.
This is Part 2. I’ll walk you through what I actually do, my own system and process for understanding if what I’ve been building has been truly useful.
Metrics alone don’t tell the full story
We stopped last time with the idea that you can’t just use a single metric to measure the value of CoDe. Time saved is easy to point to, but what about reducing risk? Or increasing the quality of work? Those are much harder to quantify. And subjective, my version of high quality work is different from yours.
In this situation, I like to think of data scientists. They have to interpret thousands of data points to create a story. Instead of focusing on a single data point, they use all of them to tell the story of the solution. It went so far that they could even use the data to predict a girl was pregnant before she herself knew
What I learnt from this, is that it’s important consider every metric to build the story of something.
So, yes measuring things is important but what you do with those numbers later is also important. But unlike applications (web or desktop), we don’t have thousands of data points to track. We have at most 2-5 quantifiable metrics.
So, I want to focus on tracking and measuring the value of scripts, not systems. Because that seems like the harder problem to solve and one that is not talked about often.
What do I actually do ?
In part 1, we looked at the metrics of: time saved, risk reduced, and quality improved. But how do we bring all together to tell the story of our scripts?
Well, it’s not perfect but I have a system that I use. It’s one that combined whatever we can measure together with notes and any other context to help tell the story.
Let me show you.
It starts with a homepage (Setting intent)
For every solution I make, I create a “homepage”. A single note where I can see everything related to it.
I start by answering a few basic questions first:
Who’s on the project? Who’s the client?
What seems to be the problem?
How do I think I’ll solve it?
Do others have to use what I build?
Out of all the metrics, which one is the most important? (time? risk? quality?)
How long do I think it will take to create?
How many people will use this?
How much time do I think it will save?
Is maintenance going to be hard?
You can modify the questions here but the general idea is to give some background information. It may seem silly to not it down, but come back in 6 months and you’ll be glad it’s there. At least, I always am because I tend to forget these things when I am not actively working on them.
Then, I try to estimate how long it will take me to create the script. (Side note: I’ve been doing this for years and I’m still terrible at it. Humans just aren’t good at estimating time needed.) Then, I estimate how much time users will save by using it.
This gives me a rough sense of the return on investment (ROI) and how many people need to use the tool to “break even.”
The more people use it after that, the more valuable it becomes and it gives me breathing room to add features and fix bugs. You can do this in Excel but with a tool like Notion, you can just punch in those numbers as the property of the page. And you can see these numbers together with the background information from before.
Here, you can add more metrics and tags that you think are important. Like if you need to know it’s Grasshopper or Dynamo or even a Python script.
Okay, but estimating and tracking are two different things. We’ve setup the homepage with some estimates but as the scripts gets developed, I’ll actually edit those numbers to reflect what is real.
If I am actively working on the script, I’ll do this once a week. If not, I’ll check the analytics and put the numbers in once a month.
Now, I should stress that they don’t have to be exact. You’re just looking for a rough return of investment here. And remember, over indexing on only time saved is a bad idea.
A note on tracking usage in computational scripts.
Tracking our own development and maintenance time is easy. We just have to remember to log it, or you can use time tracking apps like Kimai to automatically log your time to certain scripts. You can even (gasp) have a timesheet of how you’re spending your time towards these scripts.
The hard part is getting usage numbers from your script users. Grasshopper or Dynamo doesn’t have native analytic tools.
So, what I do, is I made my own. I made a plugin in Grasshopper / Dynamo that send some information to Google Analytics. It’s a plugin that I would like to release to the public soon for free but it wasn’t too hard to make if you know C#.
All I am really sending is the username and the name of the script they opened. That way I know what sticks and what doesn’t.
Adding Notes (The context)
If we don’t have that many data points to use, we have to rely more on our notes to tell the full story. If you use something like Notion, you can actually link other notes back to the homepage. This seems small but if you’re consistent, it gives you the history of the script. It makes “let me look back at my notes” way easier.
The point of capturing the history here is to understand the context of the script. Why you made it. What has happened since. And was is actually valuable. We are now looking to communicate the risk reduction and improving quality aspect of the script.
Computational scripts can solve both depth problems and scale problems. We need a way to capture both. ROI on time saved is good for problems at scale but is a horrible metric for depth problems.
So the context of the script is important here.
Did the script solve a problem impossible my hand? Did it unlock a new way of seeing things? Did it give more clarity?
All of these questions indicate value that the metric of time saved doesn’t.
Having all the notes, the background information and updated usage numbers gives me a good sense if the script was actually useful or not. Even without going through each note, a high number of notes relating to the script means there was a lot of attention on it. Which signals that people are interested in it’s functionality.
Again, I stress that this doesn’t have to be perfect. It’s a way of finding out if what you’ve built was useful or not.
Okay, you can technically stop here. But if you’ve done this enough, you know that most scripts aren’t made in isolation. They can be variants from older scripts. For me, it’s worth knowing that. It’s worth knowing the contributions that older scripts made to your new ones. Because, it may not be valuable to the users, but it was to you.
Tracking Contributions (Value beyond usage)
To do this, I make use of Notion’s linking feature again. I can now link together previous scripts to the homepage.
It may not sound like much but remember the whole point here is to get the full story of the script. So I want to know everything related to it.
Ideal vs. reality
The ideal version of this system is that I track everything perfectly and everything is automated .Every hour logged, every decision documented, every reuse captured and broken down into exact dollar figures.
While I dream of that in my sleep, reality is way messier than that. I’ll forget to update the numbers. Sometimes a solution gets reused and I don’t find out until much later. Sometimes my analytics doesn’t work and I don’t get the real usage numbers,
But the point of this system, as imperfect as it is, gives me more clarity than trusting my memory. And when I do come back to a script months later, I’m always glad the notes exist because it at least informs me of it’s value.
Final Thoughts
The idea of measuring and capturing value is a hard task for anything. It’s especially hard for tools that don’t support it (like Grasshopper or Dynamo)
But any system (even manual and incomplete ones) that gives you clarity is better than nothing. Often, your memory is clouded by how the project felt. If things didn’t go well, you might think it wasn’t good, even if it was actually useful in the end.
The idea behind my system is a mix of metrics and context to tell a fuller story of each script. I started it because I wanted to know if I was making useful things for people and it at least tells me, to some degree, the impact of my work.
The goal isn’t a perfect accounting of every hour and every use. It’s having enough information to make better decisions about what to build next, what to maintain, and what to let go.
I know tracking value sounds like overhead when you’re already busy building things. But just relying on memory and feelings doesn’t serve you well in the long run. You don’t need a perfect system. You just need a system.
You can build your own in Notion or if you’re a paid member of CodedShapes, grab the template below and skip the setup.
Get the Notion Template (Paid Members)
If you are a paid member of CodedShapes, you get the Notion template that I have shown here today. It’s something you can duplicate and start using immediately.
