OP there are two reasons people usually buy small machines, one they think they are cheaper. #2 and what you have focused on is shop space.
Cheaper eh, maybe but often not. If you have the space for large machines there are some great deals out there, so a small budget is not necessarily a good reason to buy a small machine.
Size, now that is something very hard to get around. This is an issue I know well. Getting creative many of us have fit much larger machines than we initially thought was possible, but your space does dictate your choice of machines. The best deal on a Bridgeport or a 20x96" lathe doesn't matter when it physically will not fit into your space.
I have a Sherline lathe and mill which are quite small by machine tool standards (the lathe is about 30" long and weighs about 30lbs, the mill stands about 2 feet tall and weighs around 50lbs). These are what I started with and I really like them, however did I mention they are small?
These two machines have been perfect for the projects I was interested in doing. They will chew right through brass, or aluminum, steel is also possible although it takes more finesse. I haven't worked anything harder, but I know of people working stainless steel and titanium so they can work hard metals, again just a matter of time. After about 2 years, I ran into a problem, my interests grew, and I found myself wanting to make bigger stuff. So I ended up getting a 1970s vintage 11x24" lathe, which was soon joined by a 1960s vintage 6x24" vertical mill. That was a challenge getting them into my little basement shop, but I did it. Even found space for a small horizontal mill to keep the vertical mill company. The holiday decorations took a trip to a storage unit, until I can build a shed for them.
Although I now have some bigger machines, I still do the majority of my work on the smaller ones.
I think the biggest problem people have when buying small machines is having unrealistic expectations. Many of these small machines are quite capable, and can make you happy so long as you understand going in that it is not a fullsize machine that is limited to small work. Small machines may lack power and rigidity, and that is fine so long as you compensate by taking smaller bites. They also lack many of the features on full size machines like a quick change gear box on a lathe, they may only have power feed on some travel, or lack it entirely. Small machines can be very accurate, although cheap machines may not be. Good small machines are not toys, but some of them kind of are. Some of the cheaper small machines will take a lot of work to make them really work as they should. The iconic Chinese 7x10-16" mini-lathe is a great example of this. You can buy the 7x12 at Harbor Freight for less than $700, you can also expect to need to do a lot of fine tning and modifying before you are happy with it (and may never be). You can also buy the same lathe from somebody who has done a lot of this work for you. Little Machine Shop is one vender who sells an upgraded 7x16 lathe, which is the same basic lathe as the one HF sells, but with a larger motor and they have already done a lot of the work improving it. It also costs $300-600 more depending on how much improvement you choose to buy.
As far as what are the best? Very subjective, but something like Precision Matthews 10x22 lathe and their PM25 7x27" mill can give you a lot of capability in a relatively small package, and runs on 120v. This is the combination Blondihacks (a youtuber) runs and it fits into 1/2 of a 2 car garage while still leaving room to park 2 cars. Both of these machines weigh around 350lbs so you can move them with help, but they aren't exactly mobile unless you put them on wheels which can bring its own set of problems. Grizzly also sells some similar sized machines.
Too big? The 8x16 and 9x20 lathes are about a foot shorter in length and lighter (about 170 for the 8x16, 250 for the 9x20) but are still a pretty good step up from the smaller mini-lathes. A big advantage over the 7x lathes is they are built to a higher standard of quality. The basic Chinese 7x lathes are built to a pretty hard price point. The 8 and 9" lathes are still built to keep the price down but they tend to run almost 2x the price, so you get more features, higher quality and more initial tooling for the money. I'm really not very familiar with the comparable mills and would probably still point you towards the PM25 or Grizzly G0704 at this size, although Grizzly and Little Machine Shop does have some smaller mills that could be worth looking at.
There are a variety of small vintage lathes from the Atlas 6" lathes which compare fairly well in size to the 8x16 and 9x20 lathes and there several that are closer in size to the 10x22 I mentioned above, 9 and 10" Southbend, Logan, and Atlas lathes being very common. Vintage mills in the PM25 size class are fewer, and tend to be costly, Rockwell and Clausing both made a 6x24" vertical knee mill (think 1/2 size Bridgeport) that are very popular with those with small shops.
At the bottom end of the size scale you have the real table top machines, the 7x lathes weighing about 100lbs, Taig and Sherline less than 1/2 that. These are machines that could literally be stored in a cabinet and used on a kitchen table (the wisdom of actually running a lathe in the kitchen is entirely up to your living situation

). Taig and Sherline are both made in the USA, and are nice little machines that run well right out of the box. They lend themselves to tinkering, and lots of owners mod them for their particular needs but they don't need a lot of work unlike the Chinese 7x lathes which usually require a fair bit of work to get them running well. These little lathes are not ideal for gunsmithing work but I know of people who have done quite a bit of work on pistols with Taig or Sherline machines. They are also fairly popular with clock and watchmakers.
So to sum up, small machines can be great tools, but you need to be realistic in your expectations or you may be disappointed. I have seen some fantastic work done on little machines, and I enjoy mine.