I've been using them for almost 4 years, so I'll expand on what was said above. I do live in a populated area with plenty of contractors and reviews, so bear this in mind.
*Not a lot of reviews in general. It's not like looking on amazon and finding 10-100 reviews on any given product. So if you're trying to find background information on a contractor, unless they put their name on AL and market themselves through there using coupons and such, you'll be lucky to find many if any reviews on a contractor.
*Quality of the reviews are all over the place.
*I'm not so certain the reviews are skewed in favor of the contractors (edited). I put up an honest account of a less than perfect experience years ago. Next business day I got a call from the contractor (who I had been unable to get off their a$$es for a few weeks). By the following week they were out at the house finishing the job. That same afternoon I updated the review to reflect was actually transpired and the following day I got a call from the contractor asking if I was happy and then asking me to update my review. When I told him I had...to reflect what actually transpired, they sounded pretty disappointed that I was unwilling to edit the review to make it seem rosey! AL never modified the review nor did they call me asking me to edit it.
*There are other sites similar to this, but some funding by the contractors that choose to advertise. If this is to benefit the consumer in any meaningful way, it can't be supported by the contractors. It ***** that you have to pay for the service, but I'd rather have that than something paid for by the contractors themselves.
*I did cancel the service (which was a PITA), then came back when a contractor offered a service I was interested in. The added cost for non-members basically meant it cost me nothing to re-enlist and get the member price. I forgot to cancel when that period was up and sent them a nasty email because they didn't send me a reminder or anything. I explicitly told them "no more automatic renewals under any circumstances". We shall see if they followed my instructions.
*I'm currently requesting quotes from contractors on a new roof. Now this is a category where there are tons of reviews in AL. Some were found on AL, some from local contacts. The best quote and the one I finally settled on was from a contractor that seems to feed off AL reviews. He had a very large number of reviews (37 or so) overwhelmingly positive. The only negative review was a person that postponed their job on the day the work was scheduled to begin. It sounds like the contractor wasn't happy, nor was the customer, and the situation never really improved after the initial goof (by the customer). Anyway, I hope this ends well, but my dealings with this person so far have been excellent, AND they don't require a 50% cash deposit like every other contractor I've spoken with. They collect payment only on completion of the job and also provide a 15 year guarantee on labor which is longer than everyone else's standard warranty (though the others are happy to upsell to an extended warranty).
*I've had no luck finding bargain rate contractors, but I've had pretty good success finding contractors that are reliable and easy to deal with. And honestly, I'll take that any day over the bargain price because it costs me too much time and energy to deal with dodgy people and shoddy work. If that's what I want, I can use craigslist and that's free.
*Membership price is a little steep. If you're doing something big like a roof, I'd say it's worth it to check out or screen your potential candidates, just don't expect every contractor to be listed there, as that will not be the case. But if you need an odd job done here and there, I'm not sure it's worth the membership price. This site is a tool, like any other. And if you don't have good referrals from friends and neighbors, it's better than nothing at all. But given the choice, I would rather rely on friends and neighbors, because I haven't found reviews in some categories of work I wanted to hire out. The site has the potential to be so much more valuable (like Amazon) but not under their current subscription model. There simply aren't enough reviews to really make it the one stop shop they would like it to be.
As for the comment about using the BBB, I think that helps you identify the really bad actors, but it does not necessarily help you identify the good contractors or differentiate them from the "okay" contractors (my opinion on the matter at least). I used that as a final sanity check to make sure there is nothing dodgy about the person I have selected, but I don't use it to come up with my short list.