It does make sense.
For instance, let's say in the future, you add a couple of switches. The rack gets filled up. Now you need to feed connect to a switch that is 3 feet lower/ higher in the rack. You just use a longer/ shorter patch cord. Besides, clips break and you will need to cut/ strip to make a new head. That means the cable will have to be terminated and tested again. Using patch panels, is just a smarter and cleaner way to do it.
If this was not a home application and was for a business or in industry, I could see a point to this. Since I believe that most of us are talking about home applications, I understand what you are saying, but I really don't see that many home networks having multiple switches, multiple servers, etc.
I did some googling and for the most part people seem to agree with the terminate the solid CAT5 wire at a punch down rather than a plug. Most of these opinions are based upon business environments that typically change on a fairly regular basis (I can see a point there as well). Some opinions were that the "patch cables" are more flexible and terminate to plug ends better than solid wire does. (I can see some merit to this as well).
Still other opinions match what my thoughts are: "In a home network, where there are few cable runs (a dozen or so), a patch panel tends to be overkill."
Now that I have a bunch of opinions and no hard and fast rules, I tend to believe that solid wire should be punched down to a terminal instead of terminated to a plug end. I still don't believe that I will end up plugging and replugging my network cabling enough to worry about plug tang breakage, but it may be a valid point.
My current plan is to use an inexpensive punch down block and use patch cables.