Product is having a moment. In this new series, The Product Moment, I chat with product managers who are leveraging the latest advancements in AI and data science, about their product philosophy and approach to building for markets that don’t necessarily exist yet.
For our second episode, I interview a Founding Member of Arta Finance, Petra Grutzik.
Petra is a Machine Learning Engineer and Product Leader who previously was a founding member of Arta Finance’s Product team. Petra is a Stanford alumni and product leader who was previously a Product Manager on Google’s GPay and ML Research teams.
Petra joins us in today’s episode to shed light on the complexities of building products at the intersection of AI and Fintech. She shares her insights on being a PM for AI research teams, and how best to overcome certain bottlenecks across AI and FinTech. Finally, Petra also shares what a product manager can learn from a volleyball player!
Watch Episode 2 now on YouTube.
Read the transcript below which has been condensed and revised for clarity:
Question 1: Background
Gabriella Garcia: Petra, thank you for being on The Product Moment today. It’s awesome to have you.
Petra Grutzik: Thank you so much. I’m glad we get to chat.
Gabriella Garcia: I would love for you to walk us through your career and product journeys. Starting wherever you want, but what led you to Arta and beyond is a great place to start.
Petra Grutzik: I started out studying computer science at Stanford. I fell in love with AI and decided to go into product after graduating.
I first went to Google Research because I wanted to work on AI-driven products. I got to work on this cool augmented reality device. After that, I wanted to try out the other end of the world of tech, so I went to Google payments, which was the most real-world thing I could think of in contrast to augmented reality. I primarily focused on this big banking app as part of Google Pay, and had a great time.
I then followed the Google SVP Caesars Sengupta over to his new company, Arta Finance. We are working on an ML portfolio management system for a digital family office. We’re now live on the app store, managing assets. I have since decided to explore some new opportunities. So I’ve stepped away, and I’m taking a beat.
Question 2: What is underhyped and overhyped in AI right now
Gabriella Garcia: Amazing Petra. I’m super curious – since you’re an ML nerd and are currently exploring the entire universe of applications of AI at the moment, what do you think is overhyped and underhyped in AI right now?
Petra Grutzik: This is a great question. It’s on the top of everyone’s mind because of the generative AI wave. The go-to response is that this wave is looking quite real. It’s exciting. In general, there are a couple of different applications of gen AI. There’s one that’s focused on the creative front, very B2C. This one has tons of players – you’re just seeing this plethora of startups in the market. I think they’re interesting, but most of them likely won’t succeed in the long term since they’re not providing enough value.
On the other hand, there are some interesting applications of LLMs in SaaS. For example, we already see summarization models and chatbots entering the customer success world.
We both have worked in fintech banking for quite some time and have seen firsthand how slow and awful these banking systems are, built 40+ years ago.
Gabriella Garcia: Yes, extremely archaic.
Petra Grutzik: Yes! Their APIs are old; banks are still sending physical checks.
Gabriella Garcia: If there even are APIs
Petra Grutzik: Exactly. If there are APIs, they likely won’t be real-time APIs, as payments are all done in batches. There is a lot of unstructured data, everything from downloaded PDFs to faxes to Excel spreadsheets, that real people have to sort through as part of most banking workflows. LLMs could process this unstructured data and automate these workflows, saving tons of time.
Banks have so far struggled to update their software previously because they often don’t have large engineering teams, and it’s complex and risky to transfer their system, considering it could be supporting hundreds of millions of dollars.
Gabriella Garcia: Billions even could be processed per day
Petra Grutzik: Right, for some banks, billions. And this problem is shared with other industries like healthcare and insurance. Giant industries came online 30 years ago and haven’t changed their backend technology since. We could place an LLM-enabled layer on top of archaic systems instead of transferring the system to a whole new service.
Gabriella Garcia: That’s amazing. I absolutely agree. There is a lot of value in delightful user experiences. For example, our profile pictures in this product series are stable boosted. However, to create an impact in multiple industries, you need to bring a more significant value proposition that encourages people to pay.
Question 3: Product across different spaces
Gabriella Garcia: Now, taking a step back, with your experience in ML research, Banking and Payments infrastructure, and now ML infrastructure within FinTech, how would you characterize a product role as different across these disciplines?
Petra Grutzik: Totally. It differs drastically by focus, domain expertise, and stakeholder interactions:
The goal of a PM on AI Research teams is to apply cutting-edge machine learning models to different product applications. You need a strong understanding of AI concepts, their strengths and limitations, so you know how they can best be leveraged in different use cases. You also have to be open to the opportunities that research innovations could unlock. You must be entirely up to date on new approaches and understand how to apply them to new use cases. Often working with AI applications is usually heavy on data. So you have to be comfortable diving into the weeds and taking your time to understand the details.
PMs working on Banking and Payments need deep knowledge of financial systems, payment processing, relevant regulations, and integration with existing banking infrastructure. You also likely work more closely with external partners financial institutions, payment processors, and regulators.
As a PM on ML Infrastructure within FinTech products, you need to understand both the technical aspects of AI and the intricacies of financial systems. You collaborate with a wide range of stakeholders, including data scientists, engineers, financial experts, compliance teams, and business stakeholders to develop, implement, and maintain AI-driven financial solutions.
In general in fintech, you need to ensure the systems reliability, security, and regulatory compliance, so you will likely work closely with legal and compliance teams.
Gabriella Garcia: Absolutely, I’ve always thought you have to be a lawyer to even talk to a lawyer! The amount of escalations within fintech is also very unique. While there are escalations across most product roles when dealing with finances, you have many failure points, Regulation updates, banking rails not working, SVB today falling apart, etc. In addition, money and finance are such a core part of how we live, so you must be extraordinarily delicate in balancing the system and product at all times.
Petra Grutzik: Right. Fintech is such high stakes, and each one of those escalations and bugs could risk millions of dollars. As a result, we’re much more cautious about launching. We focus on implementing safety systems and slowly ramping up products to ensure we don’t face a significant escalation in the future. But of course, escalations happen. So as a PM, you need protocols to deal with those failures and ensure you can resolve them as quickly as possible.
Question 4: How to be a strong AI/ML PM
Gabriella Garcia: I’m kind of curious, how would you recommend other folks become strong ML or AI, PMs? What are some things that you’ve learned over time that are really helpful?
Petra Grutzik: The AI PM role is quite under-defined at this point. It can differ drastically, depending on where you’re working. There are some roles in which the AI mechanisms are pretty abstracted from the applications, so you don’t need to know much about AI. You just need to focus on being a good PM and building something that has a lot of value for the user.
In other applications, you do need to go into the weeds. You need to have a much better understanding of the type of AI you’re working on, whether that is generative AI, computer vision, NLP, LLMs, etc. You need to be comfortable diving into the data to make nuanced decisions.
Question 6: Unique Challenges that AI PMs face
Gabriella Garcia: On that point, what are some unique challenges that AI PMs face?
Petra Grutzik: There are a couple of important ones. First, you must ensure that what you’re working on needs AI. There are a lot of problems that we can just solve with a rule-based or heuristics approach. AI is fascinating and sexy, but it can be slow to develop. It is much cheaper to do a rule-based approach if you can get away with it.
Second, AI applications don’t provide perfect results every single time. You’re not gonna get 100% effectiveness or reliability. You may get the algorithm to perform as expected 95% of the time. Therefore, you have to be comfortable understanding your success metrics and error bounds and knowing what 95% efficacy means for the end-user experience.
AI isn’t just servicing a button. You’re using AI in a more nuanced context, like language, computer vision, or the stock market. Therefore, the decisions you make on the end-user experience is a more complex output. It might be a full sentence, or it might be a picture, or a decision based on that picture. You won’t necessarily receive a binary output, yes or no, from a user on whether that experience is successful. You might get a “yes, that sentence sounds pretty good,” but as a PM, you have to decide what is good enough and what makes a great experience.
You also need to be mindful of staging product feature development in sync with research progress. You can add features iteratively since each component could take quite a bit of research and might not end up even working in the end.
You also have to keep monitoring your algorithms’ performance once you’ve launched. You might see a different performance in production than in testing because the algorithm is seeing new data for the first time that you didn’t train on. So, you likely need to make adjustments as you go. This could be building a reinforcement loop to retrain the models on the edge cases your model errors on. In addition, you need to be comfortable shutting down the product if it isn’t performing as well as expected in risky situations. A PM can come in to define what success is and help the research team prioritize the correct bugs to fix.
Question 6: Bottlenecks within AI x FinTech
Gabriella Garcia: Amazing – returning to the intersection of AI and FinTech, what are some of the main bottlenecks for success or development?
Petra Grutzik: Totally. For the past year and a half, I’ve worked on AI applied to trading on the stock market. Data is by far the most significant bottleneck and it’s unique for a couple of reasons.
First, the stock market data is non-stationary. Many problems we solve with AI involve stationary data. For example, image classifiers deal with stationary data because the correct classification answer will always be the same; a dog will always look like a dog. However, financial markets fluctuate as a reflection of what is happening in the real world today. Therefore the correct answer for the last 50 years might not be the correct answer today. For instance, we didn’t have a real estate crash for 50 years, and then in 2008, we did. And so you can’t necessarily count on your historical data to represent the future.
In addition, there are adversarial actors within the stock market, so acting rationally might not be the best approach. For example, with GameStop, people kept buying and buying, and it kept on going up, even though, rationally this was not a good thing for people to do. Markets can be irrational far longer than individual players can be solvent. Therefore you need to have strong risk management in place to ensure your AI systems perform well and that you are taking the necessary precautions with your implementation.
The last bottleneck is around the limits of sanitizing incoming data. Data in general, is usually very messy. A huge part of an ML engineer’s job is to clean and structure that data. For example, you might want to make your data set cleaner by training on stocks that all have exactly 10 years’ worth of historical data. Do get this clean dataset, you could eliminate all of the companies that failed since they don’t have a full 10-year history. But you can’t go back and retroactively apply that knowledge to clean your data, because in doing so, your model would inherently have a sense of the future. You have to train your model to only base its decisions on historical events.
Gabriella Garcia: Interesting; not only could you be cleaning out potential failure cases, but also with not-perfectly-structured data. We see this a lot with climate data. For certain geographics, we can be limited by not having enough historical data points. So you can have a mix of smaller and longer data sets on the same problem you’re trying to train and test on.
Question 7: Is consensus always a good thing?
Gabriella Garcia: Let’s talk a little bit more about leadership within the PM role. Is consensus always a good thing?
Petra Grutzik: Great question. Consensus can be powerful because it means everyone’s bought into the idea. Frequently, if the team is bought in, everyone will be motivated to work in the same direction. However, there might be times when it takes a long time to drive everyone to a consensus. So it’s a bit slower of a process.
There are also times when not everyone will come to a consensus. Ideally, you disagree and commit. However, if that isn’t happening, you may need to influence management to push things forward with a top-down decision.
Question 8: What can a PM learn from a volleyball player?
Gabriella Garcia: So this is my favorite question! Petra, you used to be a competitive volleyball player, and overall, you value yourself as an athlete. What can a product manager learn from a volleyball player?
Petra Grutzik: I do love this question! I was a setter all growing up. The setter is the person who gets the second touch on the ball and tries to give a perfect set to the hitter, the person who spikes the ball on the other side of the net to score the point. Setting is very similar to being a PM.
Frequently, the setter is not the one scoring points. Instead, you identify and set the hitter who is most likely to get that kill. Like PMs, setters focus on making strategic decisions. You’re trying to set up the shots so the team can get the biggest wins by launching the best products.
Further, setters are the transition between the defense, the people receiving the ball from the other team, and the offense, the hitters. A setter’s main goal is to get the team work together as best it can to win the game. Similarly, PMs help all sorts of functions, like eng, UX, and marketing, work together seamlessly and efficiently to build the best products possible.
No matter what, like the setter, the PM strives to set up every single person on the team to play the best they can, celebrate their successes, and be there for the team’s losses and hard times.
Gabriella Garcia: Wow, that was beautifully said, Petra! This was an absolute blast having you here at The Product Moment.
Petra Grutzik: Thank you so much for having me. This was super fun! I’m excited to chat again soon.
If you want to talk about all things product, you can find Gaby on LinkedIn.
