I had this thought and implemented it months ago. However, recently I decided to use the backend I had built on another project but remove the providermanager and hardcode OpenAI as the provider. This was to make the application more lightweight as the client didn’t “need” other providers🙄. I then shipped the product, 2 days later the debacle hit and we had serious outages across many of our apps, however most switched to other providers automatically. The only one that was mostly unavailable…. You guessed it…. The one without the ability to use multiple LLM providers.
Really interesting point about the "multi-model strategy". I think it's super relevant now, that ton of small 'startups' depend 100% on OpenAI.
Regarding the abstractions - here I tend to disagree. I think that shortly being a PM will be enough to get a working product to market. You'll learn to tweak the LLM, and as they get better - they'll fix the nasty bugs themselves (you'll just describe the bug). Currently engineers do have an advantage, as they can 'fill-in' the holes where needed, but I believe it's temporary.
PMs are really prompt engineers for developers - they know how to talk to us in our language, they give us Figma, user stories, and so on. It will be the same, but with LLMs :)
I had this thought and implemented it months ago. However, recently I decided to use the backend I had built on another project but remove the providermanager and hardcode OpenAI as the provider. This was to make the application more lightweight as the client didn’t “need” other providers🙄. I then shipped the product, 2 days later the debacle hit and we had serious outages across many of our apps, however most switched to other providers automatically. The only one that was mostly unavailable…. You guessed it…. The one without the ability to use multiple LLM providers.
Really interesting point about the "multi-model strategy". I think it's super relevant now, that ton of small 'startups' depend 100% on OpenAI.
Regarding the abstractions - here I tend to disagree. I think that shortly being a PM will be enough to get a working product to market. You'll learn to tweak the LLM, and as they get better - they'll fix the nasty bugs themselves (you'll just describe the bug). Currently engineers do have an advantage, as they can 'fill-in' the holes where needed, but I believe it's temporary.
PMs are really prompt engineers for developers - they know how to talk to us in our language, they give us Figma, user stories, and so on. It will be the same, but with LLMs :)