by finnlobsien1 on 10/10/2023, 5:08:22 PM with 1 comments
I was reading some of the posts on this subreddit about AI and whether PMs need to be worried about AI taking their jobs.
Let’s be real: No product manager will be replaced by ChatGPT tomorrow. But what if they were? I tried—by asking it to PM a product launch.
It was… weird. Some things went well, others worse.
The setup
I used ChatGPT (4) custom instructions. They’re long & detailed so I won’t share them in full (happy to post if anyone cares).
The gist of what I told it:
We’re a user assistance startup that offers tools to product people which make software easier to use. We’re launching a new AI assistant called Copilot.
You have a sales team, design team and marketing team at your disposal.
Respond like a product manager: Ask clarifying questions, focus on the goal, communicate with multiple teams, etc.
Here are the results:
What worked
Basics: ChatGPT understood how to bring a feature to market. It was also good at setting obvious goals: Target audience appeal, seamless integration, UX improvement, etc.
Those are good, but which product do we not want to appeal to our target audience or offer UX improvements?! ChatGPT’s goals were too generic to be useful. A bit like a salesperson saying their goal is “make money” lol
User stories: I asked it to formulate user stories, which it did a good job on (even if most examples were straight from the instructions).
Briefings, PRDs, documentation: I asked for a variety of briefings for marketing, design, etc. Those were good: Actionable and brief.
What didn’t work
Conflict resolution: I told it that the eng team was at my office with pitchforks and torches about the short timeline and felt like they’re doing repetitive work. GPT’s response detailed the intent to set up a meeting, but didn’t ask about underlying issues.
Communication: It never stood its ground or pushed back. When I tested it with harsh feedback, it always apologized, said I was right and vowed to correct its mistake—even if that feedback contradicted something I said earlier.
Problem-solving: To test PM-GPT’s problem-solving skills, I threw complex-ish customer feedback at it: An enterprise customer wondering how access permissions would be reflected in the chatbot’s output.
My AI PM then recommended an RBAC system and included generic instructions. It labeled the solution correctly, but that was it. It was basically the thing engineers often hate PMs for—attaching a smart-sounding framework and acronym to everything, dishing out some to-dos and calling it.
What does this mean for the future of PMs?
ChatGPT was the caricature of a bad PM: It adds acronyms and frameworks to obvious things and assigns out work without making decisions or doing anything of its own initiative.
But while it was a bad product manager, it didn’t fail at the tasks or completely misinterpret them.
Part of its failure was simply a lack of data. It’s trained on general information, so the output is general information. Had it had access to our Notion, Slack etc. its outputs would’ve been more specific. With API access, an all-seeing GPT-4 could’ve been much better.
But most of the value of a good product manager adds isn’t typing out PRDs or briefing the marketing team. It’s in managing stakeholders, interviewing users and finding novel ways to solve for their needs. The things GPT sucked at!
AI isn’t taking those from us (yet). But the logistical things ChatGPT was good at were things often done by juniors/associates/interns or simply mediocre PMs.
I think that while AI makes good PMs more powerful (while they keep their job), it might eliminate some of the junior roles, or cause one product manager to do the work of 3.
Thought I'd share this insight—happy to answer questions and share screenshots of output, specific prompts, custom instructions or a longer writeup!
Let’s be real: No product manager will be replaced by ChatGPT tomorrow. But what if they were? I tried—by asking it to PM a product launch.
It was… weird. Some things went well, others worse.
The setup
I used ChatGPT (4) custom instructions. They’re long & detailed so I won’t share them in full (happy to post if anyone cares).
The gist of what I told it:
We’re a user assistance startup that offers tools to product people which make software easier to use. We’re launching a new AI assistant called Copilot.
You have a sales team, design team and marketing team at your disposal.
Respond like a product manager: Ask clarifying questions, focus on the goal, communicate with multiple teams, etc.
Here are the results:
What worked
Basics: ChatGPT understood how to bring a feature to market. It was also good at setting obvious goals: Target audience appeal, seamless integration, UX improvement, etc.
Those are good, but which product do we not want to appeal to our target audience or offer UX improvements?! ChatGPT’s goals were too generic to be useful. A bit like a salesperson saying their goal is “make money” lol
User stories: I asked it to formulate user stories, which it did a good job on (even if most examples were straight from the instructions).
Briefings, PRDs, documentation: I asked for a variety of briefings for marketing, design, etc. Those were good: Actionable and brief.
What didn’t work
Conflict resolution: I told it that the eng team was at my office with pitchforks and torches about the short timeline and felt like they’re doing repetitive work. GPT’s response detailed the intent to set up a meeting, but didn’t ask about underlying issues.
Communication: It never stood its ground or pushed back. When I tested it with harsh feedback, it always apologized, said I was right and vowed to correct its mistake—even if that feedback contradicted something I said earlier.
Problem-solving: To test PM-GPT’s problem-solving skills, I threw complex-ish customer feedback at it: An enterprise customer wondering how access permissions would be reflected in the chatbot’s output.
My AI PM then recommended an RBAC system and included generic instructions. It labeled the solution correctly, but that was it. It was basically the thing engineers often hate PMs for—attaching a smart-sounding framework and acronym to everything, dishing out some to-dos and calling it.
What does this mean for the future of PMs?
ChatGPT was the caricature of a bad PM: It adds acronyms and frameworks to obvious things and assigns out work without making decisions or doing anything of its own initiative.
But while it was a bad product manager, it didn’t fail at the tasks or completely misinterpret them.
Part of its failure was simply a lack of data. It’s trained on general information, so the output is general information. Had it had access to our Notion, Slack etc. its outputs would’ve been more specific. With API access, an all-seeing GPT-4 could’ve been much better.
But most of the value of a good product manager adds isn’t typing out PRDs or briefing the marketing team. It’s in managing stakeholders, interviewing users and finding novel ways to solve for their needs. The things GPT sucked at!
AI isn’t taking those from us (yet). But the logistical things ChatGPT was good at were things often done by juniors/associates/interns or simply mediocre PMs.
I think that while AI makes good PMs more powerful (while they keep their job), it might eliminate some of the junior roles, or cause one product manager to do the work of 3.
Thought I'd share this insight—happy to answer questions and share screenshots of output, specific prompts, custom instructions or a longer writeup!