I never released a tool or library so the discussion is based on my experience as a library consumer.
Freya Holmér recently released Shapes, a vector graphics library for Unity, on her Patreon. She invited patrons who asked to her repository, and it went bad.
This is terrible because I think that sharing the source code with of library is essential, even for a paid one.
Even if you wish to publish your code / asset / library to the Assets Store, nothing prevent users to take your code and put it somewhere else for free.
Your code will be plain text, free for purchasers to read, modify, share, etc ... (and even if you only give a DLL and not the full code, they can still share it)
Once we establish that we can't fully control the usage of our published library comes the question: What is the best way to make money with my tool and what is the best way to work with my users ?
If you make an open source project, you could:
- Count on tips
- Make a Patreon (or similar) like the Godot team
- Sell support, RedHat mostly works that way
- Make bounties, i.e. features that some users/companies will pay you to develop
- Find sponsors, like Oculus for VRTK
And if you want to make it paid, you could put your library behind a paywall in addition to the previous options (like a Patreon or the Unity Asset Store)
I rarely see the bounty solution but I find it very interesting. I think people are more prone to donate if they want something to happen (a feature).
You just have to look at the popularity of kickstarters nowadays to see that this economic model can work.
It benefit you and the community.
One thing I hate about the Assets Store is that most of the packages there don't have a way to open a ticket or a forum to discuss about issues and solutions.
Often, your only choice is to leave a message on the Assets reviews section.
The Holmér choice to give people access to the Github repository is interesting.
Once you've sold (or shared) your code, it's important to give users a tool with which they can give you feedback, report bugs and help you to resolve these issues.
They want, as much as you do, to have a working bug-less code.
And it's way easier for them to help to track down the bug or suggest a fix if they have complete access to the code.